UI Tool:

  • https://github.com/obsidiandynamics/kafdrop
  • https://github.com/yahoo/CMAK
  • https://dzone.com/articles/kafka-administration-and-monitoring-ui-tools

PubSub VS Message Queue

  • Message Queue = Same consumer group = Competing Consumer pattern
  • PubSub = Different consumer group = Publish Subscribe pattern


ISR = In Sync Replication

Kafka Consumer Lag = Latest Offset - Consumer Offset

Burrow: Kafka Consumer Lag Checking


Get kafka PID

ps ax | grep -i 'kafka' | grep java | grep -v grep | grep -v zookeeper | awk '{print $1}'

Remove all data on topics

rm -rf /tmp/kafka-logs/newTopic-*

Create topic

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Delete topic

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic test

List of topics

bin/kafka-topics.sh --list --zookeeper localhost:2181

Server Properties

Allowed kafka to be access from outside


Get error dial tcp: lookup YOUR_NETWORK_NAME: no such host if consumer service using localhost or as host.


Allow to delete topic


Kafka Manager


git clone https://github.com/yahoo/kafka-manager.git
cd kafka-manager
./sbt clean dist


Brokers Skewed = number of partitions > avg partitions per broker (on the given topic). e.g: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2)

Brokers Spread = percentage of brokers in the cluster that has partitions for the given topic. e.g. 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic


Consume without group id

  • set enable.auto.commit to False
  • set the TopicPartition .offset to OFFSET_BEGINNING, OFFSET_END or an absolute offset to start consuming from - the default is to used the committed offset.