Kafka commands

  • Home
  • /
  • Kafka commands
Kafka commands

Kafka commands

Kafka 22 May 2018 Siva Nadesan
  • Kafka home directory is /usr/hdp/current/kafka OR /usr/hdp/current/kafka-broker
  • All kafka scripts are located in /usr/hdp/current/kafka/bin OR /usr/hdp/current/kafka-broker/bin
Kafka - Status

Command : kafka status

post thumb
Kafka - Create topic

Command : ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 13 --topic entechlog_test0001

post thumb
Kafka - List topic

Command : kafka-topics.sh --list --zookeeper localhost:2181

post thumb

You can also use kafka-topics command with a properties file when connecting a kafka broker secured by password. To do this first create a properties file like the below one and then issue the kafka-topics command.

retries=3
retry.backoff.ms=500
batch.size=65536
bootstrap.servers=localhost:9092
ssl.endpoint.identification.algorithm=https
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="your-user-name" password="your-password";

Command: kafka-topics --command-config config.properties --list --bootstrap-server localhost:9092

Kafka - List topics with overrides

Command : kafka-topics --zookeeper localhost:2181 --describe --topics-with-overrides

Kafka - Describe topics

Command:  kafka-topics --describe --zookeeper localhost:2181 --topic "connect-offsets"
OR
Command : kafka-configs --zookeeper localhost:2181 --describe --entity-type topics --entity-name "connect-offsets"

Kafka - Delete topics

Command : kafka-topics --delete --topic dev.etl.datagen.user.src.0010 --zookeeper localhost

If you don’t have delete.topic.enable=true enabled, Then delete will have no effect.

Kafka - Describe brokers

Command : kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --describe

Kafka - Alter topics

Command : kafka-topics --zookeeper localhost:2181 --alter --topic connect-offsets --config retention.ms=-1
OR
Command : kafka-configs --zookeeper localhost:2181 --alter --entity-type topics --entity-name "connect-offsets" --add-config retention.ms=-1

Kafka - Alter brokers

Command : kafka-configs --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config log.retention.ms=604800000

Kafka - Find the size of Kafka topic

Command :  kafka-log-dirs --describe --bootstrap-server localhost:9092 --topic-list connect-offsets

Kafka - Command to count the number of records in a topic

Command :  kafka-run-class kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic connect-offsets --time -1 | tr ":" " " | awk '{ sum += $3 } END { print sum }'

Kafka - Console producer

Command : ./kafka-console-producer.sh --broker-list localhost:6667 --topic entechlog_test0001
OR
Command : ./kafka-console-producer.sh --broker-list localhost:6667 --topic entechlog_test0001 --security-protocol SASL_PLAINTEXT
OR
Command : kafka-console-producer --broker-list localhost:9092 --topic test-topic-console-producer-01 --producer.config /path/producer.properties --property "parse.key=true" --property "key.separator=:"

Kafka - Console consumer

command :  ./kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic entechlog_test0001
OR
Command : ./kafka-console-consumer.sh --bootstrap-server localhost:6667 --topic entechlog_test0001 --from-beginning
OR
Command : ./kafka-console-consumer.sh --bootstrap-server localhost:6667 --topic entechlog_test0001 --from-beginning --security-protocol SASL_PLAINTEXT
OR
Command : kafka-avro-console-producer --broker-list localhost:9092 --topic test-topic-avro-producer --producer.config /path/producer.properties --property "schema.registry.url=https://localhost" --property schema.registry.basic.auth.user.info="your-user-name:your-password" --property basic.auth.credentials.source="USER_INFO" --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"id","type":"int"},{"name":"product", "type": "string"}, {"name":"quantity", "type": "int"}, {"name":"price","type": "float"}]}'

Last example uses a schema so you should pass a message which is compatible with the schema like the below one

{"id": 999, "product": "foo", "quantity": 100, "price": 50}
post thumb
Kafka - Test Connection to Kafka broker
  • Test using netcat command nc -vz localhost 9092
  • Test using curl command curl -v localhost:9092
  • Test un-secured endpoint using curl command curl -v https://localhost
  • Test secured endpoint using curl command curl -v -u "<API_KEY>:<TOKEN>" https://localhost


About The Authors
Siva Nadesan

Siva Nadesan is a Principal Data Engineer. His passion includes data and blogging about technologies. He is also the creator and maintainer of www.entechlog.com

LinkedIn

Share: