Kafka

• How to choose partition number
• Replication Tool
• Kafka Authorizer
• Don’t use consumer groups the wrong ways

UI Tool:

• https://github.com/obsidiandynamics/kafdrop
• https://github.com/yahoo/CMAK
• https://dzone.com/articles/kafka-administration-and-monitoring-ui-tools

PubSub VS Message Queue

• Message Queue = Same consumer group = Competing Consumer pattern
• PubSub = Different consumer group = Publish Subscribe pattern

Lag

ISR = In Sync Replication

Kafka Consumer Lag = Latest Offset - Consumer Offset

Burrow: Kafka Consumer Lag Checking

Commands

Get kafka PID

ps ax | grep -i 'kafka' | grep java | grep -v grep | grep -v zookeeper | awk '{print $1}'

Remove all data on topics

rm -rf /tmp/kafka-logs/newTopic-*

Create topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Delete topic

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test

List of topics

bin/kafka-topics.sh --list --zookeeper localhost:2181

Server Properties

Allowed kafka to be access from outside

advertised.host.name=0.0.0.0

Get error dial tcp: lookup YOUR_NETWORK_NAME: no such host if consumer service using localhost or 127.0.0.1 as host.

listeners=PLAINTEXT://:9092 

Allow to delete topic

delete.topic.enable=true

Kafka Manager

Install

git clone https://github.com/yahoo/kafka-manager.git
cd kafka-manager
./sbt clean dist

ActorModel.scala

Brokers Skewed = number of partitions > avg partitions per broker (on the given topic). e.g: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2)

Brokers Spread = percentage of brokers in the cluster that has partitions for the given topic. e.g. 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic

Consumer

Consume without group id

• set enable.auto.commit to False
• set the TopicPartition .offset to OFFSET_BEGINNING, OFFSET_END or an absolute offset to start consuming from - the default is to used the committed offset.