Setup kafka on linux

#make sure ubuntu is updated
sudo apt-get update -y
sudo apt-get upgrade -y

#install java
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer -y
check:
sudo java -version

#install zookeeper
sudo apt-get install zookeeperd
check :
netstat -ant | grep :2181

#install kafka server
http://kafka.apache.org/downloads

unzip files to /opt/kafka

#start kafka server :
sudo /opt/Kafka/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh /opt/Kafka/kafka_2.11-0.11.0.0/config/server.properties

#create topic :
sudo /opt/Kafka/kafka_2.11-0.11.0.0/bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic testing

#list topic :
sudo /opt/Kafka/kafka_2.11-0.11.0.0/bin/kafka-topics.sh –list –zookeeper localhost:2181

#produce message:
sudo /opt/Kafka/kafka_2.11-0.11.0.0/bin/kafka-console-producer.sh –broker-list localhost:9092 –topic testing

#consume message:
udo /opt/Kafka/kafka_2.11-0.11.0.0/bin/kafka-console-consumer.sh –zookeeper

localhost:2181 –topic testing –from-beginning

 

 

Python sample :

<br />from kafka import KafkaConsumer,KafkaProducer

def run():
print('Step1');
consumer = KafkaConsumer(bootstrap_servers='localhost:9092', auto_offset_reset='earliest')
print('Step2');
consumer.subscribe(['testing'])
print('Step3');
for message in consumer:
print (message)
print('Step4');

def produce1():
print('prducing 1');
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('testing', 'message from producer.').get(timeout=30)

produce1();
run();

 

 

Author: lanliang

Programmer.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s