In this article we will discuss some of the common Confluent Kafka challenges and solutions to fix them.

Issue 01

Issue Running a select statement in KSQL CLI gives java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.DBOptions

Solution
Set the KSQL_OPTS and then bring up confluent KSQL

cd /opt/confluent
mkdir javatmp
export KSQL_OPTS="-Djava.io.tmpdir=/opt/confluent/javatmp/"
confluent local start

Assumptions

• You have installed confluent platform using the tar.gz
• You have installed confluent platform in /opt/confluent

Issue 02

Issue Unable to issue confluent local start because a previous instance is running

Solution

• Stop all current services using confluent local stop and then issue confluent local start
• If you still unable to issue confluent local start, Then find ps -ef | grep confluent and kill all confluent processes which is currently running and then do a confluent local start

Issue 03

Issue Control center won’t stay up because /tmp is not mounted with execute access

Solution Set the CONTROL_CENTER_OPTS and then bring up confluent control center

cd /opt/confluent
mkdir rocksdbtmp
export CONTROL_CENTER_OPTS="-Djava.io.tmpdir=/opt/confluent/rocksdbtmp"
confluent local start

This can happen every time you restart your server/service, So avoid this happening you can set your /etc/profile and /etc/bashrc to have following export statements. If you are on Ubuntu distribution then you may need to update /etc/bash.bashrc and /etc/profile.d/myenvvars.sh.

export CONFLUENT_HOME=/opt/confluent
export CONFLUENT_CURRENT=${CONFLUENT_HOME}/var
export PATH=${CONFLUENT_HOME}/bin:${PATH}
export CONTROL_CENTER_OPTS="-Djava.io.tmpdir=/opt/confluent/rocksdbtmp"
export KSQL_OPTS="-Djava.io.tmpdir=/opt/confluent/javatmp/"

Issue 04

Issue
Confluent systemctl services confluent-kafka, confluent-kafka-rest, confluent-schema-registry, confluent-ksql fails when they start up.

Solution
This happens when these services are started before zookeeper is up and running, edit the systemctl and add the following dependencies and restart on failure option.

sudo systemctl edit --full confluent-kafka
sudo systemctl edit --full confluent-kafka-rest
sudo systemctl edit --full confluent-schema-registry
sudo systemctl edit --full confluent-ksql

[Unit]
Requires=confluent-zookeeper.service confluent-kafka.service
After=confluent-zookeeper.service confluent-kafka.service

[Service]
Restart=on-failure
RestartSec=3
Environment=KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=localhost"

Note
Requires and After entries for confluent-kafka.service is not needed in confluent-kafka systemctl

• Do a sudo systemctl restart confluent* and sudo systemctl status confluent* to validate services are working after this change.

Assumptions

• You have installed confluent platform using systemd on Ubuntu and Debian

Issue 05

Issue
Confluent systemctl service confluent-ksql fails during startup with “Could not create the kafka streams state directory: /var/lib/kafka-streams”.

Solution

• Make sure the directory exists and has right permissions OR
• Create a new directory say /opt/confluent/javatmp/ and update /etc/ksql/ksql-server.properties to point to this new directory ksql.streams.state.dir=/opt/confluent/javatmp/

Note

• Do a sudo systemctl restart confluent-ksql and sudo systemctl status confluent-ksql to validate services are working after this change.
• You can also export full systemctl service logs to a file using journalctl -u confluent-ksql.service > /tmp/confluent-ksql.service.log.$(date "+%Y.%m.%d-%H.%M.%S")

Assumptions

• You have installed confluent platform using systemd on Ubuntu and Debian