Kafka cluster Getting Started

kafka는 분산 메시징 시스템으로 높은 성능을 자랑한다.

A high-throughput distributed messaging system.

분산 시스템을 구성 위해서 3개 노드에 zookeeper, kafka를 모두 설치한다.

Download

http://kafka.apache.org/downloads.html

Setup ZooKeeper

$ vi ${KAFKA_HOME}/config/zookeeper.properties

zookeeper.properties 파일에 아래 설정들을 추가해준다.

# the directory where the snapshot is stored.

dataDir=${data_dir}

# The number of milliseconds of each tick
tickTime=2000

# The number of ticks that the initial
# synchronization phase can take
initLimit=10

# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5

server.1=${zookeeper_server_1}:2888:3888

server.2=${zookeeper_server_2}:2888:3888

server.3=${zookeeper_server_3}:2888:3888

각 서버에 위에서 설정한 myid 값을 설정한다.

${zookeeper_server_1}:$ cat > /${data_dir}/myid

1

${zookeeper_server_2}:$ cat > /${data_dir}/myid

2

${zookeeper_server_3}:$ cat > /${data_dir}/myid

3

run ZooKeeper

$ bin/zookeeper-server-start.sh -daemon config/zookeeper.propertie

Setup Kafka

$ vi ${KAFKA_HOME}/config/server.properties

server.properties 파일에 아래 설정을 변경한다.

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1 # 서버마다 고유한 아이디 값을 입력한다.

# A comma seperated list of directories under which to store log files
log.dirs=${kafka_log_dir}

# The default number of log partitions per topic. More partitions allow greater

# parallelism for consumption, but this will also result in more files across

# the brokers.

num.partitions=10 # 토픽이 생성될뙈 기본적으로 몇개 partition을 생성할지 (각 시스템 별로 설정)

# root directory for all kafka znodes.
zookeeper.connect=${zookeeper_server_1},${zookeeper_server_2},${zookeeper_server_3}

run Kafka

$ ./bin/kafka-server-start.sh -daemon config/server.properties

Zookeeper Client Test

# 주키퍼 테스트
./bin/zookeeper-shell.sh ${zookeeper_server_1},${zookeeper_server_2},${zookeeper_server_3}

Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
ls /
[consumers, config, controller, admin, brokers, zookeeper, controller_epoch]   <= 목록이 나오면 kafka까지 정상

Kafka Test

토픽을 생성하고 예제 producer와 consumer로 테스트를 진행한다.
# 토픽 생성
> bin/kafka-topics.sh --create --zookeeper ${zookeeper_server_1},${zookeeper_server_2},${zookeeper_server_3} --replication-factor 3 --partitions 10 --topic my-replicated-topic
Created topic "my-replicated-topic".

#  토픽 목록 조회 
./bin/kafka-topics.sh --list --zookeeper ${zookeeper_server_1},${zookeeper_server_2},${zookeeper_server_3} 
test

# 토픽 정보 확인 
> bin/kafka-topics.sh --describe --zookeeper opdev-infra01:22181,opdev-infra02:22181,opdev-infra03:22181 --topic my-replicated-topic
Topic:my-replicated-topic	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: my-replicated-topic	Partition: 1	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: my-replicated-topic	Partition: 2	Leader: 1	Replicas: 1,3,2	Isr: 1
	Topic: my-replicated-topic	Partition: 3	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1
	Topic: my-replicated-topic	Partition: 4	Leader: 3	Replicas: 3,1,2	Isr: 3,1,2
	Topic: my-replicated-topic	Partition: 5	Leader: 1	Replicas: 1,2,3	Isr: 1
	Topic: my-replicated-topic	Partition: 6	Leader: 2	Replicas: 2,1,3	Isr: 2,1,3
	Topic: my-replicated-topic	Partition: 7	Leader: 3	Replicas: 3,2,1	Isr: 3,2,1
	Topic: my-replicated-topic	Partition: 8	Leader: 1	Replicas: 1,3,2	Isr: 1
	Topic: my-replicated-topic	Partition: 9	Leader: 2	Replicas: 2,3,1	Isr: 2,3,1

# sample producer 실행
> bin/kafka-console-producer.sh --broker-list ${broker_server_1},${broker_server_2},${broker_server_3} --topic my-replicated-topic
test
test message

# sample consumer
> ./bin/kafka-console-consumer.sh --zookeeper ${zookeeper_server_1},${zookeeper_server_2},${zookeeper_server_3} --topic test
test
test message

참고

http://kafka.apache.org/documentation.html#introduction

http://kafka.apache.org/downloads.html