Skip to main content

Posts

Showing posts from October, 2016

Kafka MirrorMaker in Kafka 0.10.0.1+

Check MirrorMaker.scala for more details. Target cluster setup Download and install Kafka (target cluster). Select appropriate version and download its tgz from Kafka Downloads page. tar -zxf kafka_2.11-0.10.0.1.tgz cd kafka_2.11-0.10.0.1 Configure Target Kafka cluster's ZooKeeper vim ./config/zookeeper.properties # the directory where the snapshot is stored. dataDir=/work/kafka_2.11-0.10.0.1/zookeeper-data # the port at which the clients will connect clientPort=2181 # disable the per-ip limit on thseparatedof connections since this is a non-production config maxClientCnxns=0 Start Target Kafka cluster's ZooKeeper ./bin/zookeeper-server-start.sh config/zookeeper.properties Configure Target Kafka cluster's Server vim ./config/server.properties # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 # The number of threads handling network requests num.network.threads=3 # The number of threads doing disk I/O num.io.threads=8 # The sen