Configuring Apache Kafka 2.13-3.5.1 (Kraft Mode) with SSL

Configuring Apache Kafka 2.13-3.5.1 (KRaft mode) with SSL involves several steps to ensure secure communication between clients and brokers. 

KRaft (Kafka Raft) mode is Kafka’s self-managed metadata system that replaces the need for Apache ZooKeeper. (Click here to learn how to set up Kafka in KRaft mode.)

To enable secure communication, SSL certificates must be created and configured on both the broker and client sides. The following article explain the complete setup process.

Creating SSL Certificates on the Broker Side

Creating SSL certificates on the broker side involves the following steps:

1. Create a directory to store all certificates using the CLI. Example:
   

   mkdir ssl-certs

2. Create a file called san.conf using the command touch san.conf under the ssl-certs directory and enter the following details in it:
   

   [req_ext]
   subjectAltName = @alt_names
   [alt_names]
   DNS.0 = nastelin.com
   IP.0 = 172.16.31.72

   Mention the required IP addresses.

3. Generate the certificates in sequence: 
   • Run the following command to generate the keystore (keystore name and password can be set accordingly)
     

     keytool -keystore kafka.server.keystore.jks -alias localhost-keyalg RSA -validity 
     1000 -genkey -storepass nastel -keypass nastel -ext 
     SAN=DNS:nastelin.com,IP:172.16.31.72

   • Generate the Certificate Authority (CA) certificate

     openssl req -new -x509 -keyout ca-key.key -out ca-cert.crt -days 1000

   • Import the CA certificate into the truststore

     keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert-file ca-cert.crt

   • Generate the certificate signing request (CSR) for the server

     keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file-server.csr-ext SAN=DNS:nastelin.com,IP:172.16.31.72

   • Sign the CSR with the CA certificate:

     openssl x509 -req -CA ca-cert.crt -CAkey ca-key.key -in cert-file-server.csr-out cert-signed-server.crt -days 1000 -CAcreateserial-passinpass:nastel -extfile san.conf -extensions req_ext

   • Import the CA certificate into the keystore:

     keytool -keystore kafka.server.keystore.jks -alias CARoot -importcert-file ca-cert.crt

   • Import the signed server certificate into the keystore

     keytool -keystore kafka.server.keystore.jks -alias localhost-importcert -file cert-signed-server.crt

4. All the above certificates will appear as shown in the image below.
   
5. Add the following details to the server. properties file present in/opt/meshiq/kafka/ kafka-2.13-3.5.1/config/kraft
   

   listeners=SSL://:9093
   advertised.listeners=SSL://172.16.31.72:9093
   inter.broker.listener.name=SSL
   ssl.keystore.location=/opt/meshiq/kafka/kafka_2.13-3.5.1/ssl-certs/kafka.server.keystore.jks
   ssl.keystore.password=nastel
   ssl.key.password=nastel
   ssl.truststore.location=/opt/meshiq/kafka/kafka_2.13-3.5.1/ssl-certs/kafka.server.truststore.jks
   ssl.truststore.password=Nastel

   These are some of the important properties that need to be mentioned in the server.properties file since we are configuring it with SSL.

6. In the scripts folder provided by meshiq, modify the KAFKA_START.sh script by adding the following lines:
   

   WORKING_DIR=/opt/meshiq/kafka/kafka_2.13-3.5.1
   export JAVA_HOME=/opt/meshiq/java/current export JRE_HOME=/opt/meshiq/java/current
   export PATH=/opt/meshiq/java/current/bin:$PATH
   
   export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true 
   -Dcom.sun.management.jmxremote.authenticate=false
   -Dcom.sun.management.jmxremote.ssl=true 
   -Djava.rmi.server.hostname=172.16.31.72
   -Dcom.sun.management.jmxremote.authenticate=false 
   -Djavax.net.ssl.keyStore=/opt/meshiq/kafka/kafka_2.13-3.5.1/ssl-certs/kafka.server.keystore.jks 
   -Djavax.net.ssl.keyStorePassword=nastel 
   -Djavax.net.ssl.trustStore=/opt/meshiq/kafka/kafka_2.13-3.5.1/ssl-certs/kafka.server.truststore.jks
   -Djavax.net.ssl.trustStorePassword=nastel -Dcom.sun.management.jmxremote.rmi.port=9993"
   
    export JMX_PORT=9992

   Ensure that the script has the correct working directory and Java path

7. After adding the lines, save the changes, and run the following shell command to start Kafka:
   

   ./KAFKA_START.sh

8. To verify if the Kafka has started and is running, execute the following command:
   

   ps -ef | grep properties

9. A series of lines can be seen, where SSL is enabled. Keystore paths and passwords are mentioned as shown in the image below. 
   
   These lines depict that the SSL is enabled for Kafka running in the Kraft mode.
10. Once the broker-side setup is complete and running, the client-side properties can be configured.

Creating SSL Certificates on the Client Side

Setting up a Kafka client with SSL involves configuring the client to securely communicate with the Kafka brokers. 

As mentioned above, now we need to setup the client-side connection to allow the client server to communicate with the broker. Follow the steps below:

1. Navigate to the ssl-certs directory which was created earlier. Since, we already have the san.conf file, we do not need to create another one.
2. It is important to note that the same ca-cert.crt and ca-key.key files are used to create the client keystore and truststore certificates.
   • • Generate the client keystore
       

       keytool -keystore kafka.client.keystore.jks -alias clientKeystore
       -validity 365 -genkey -keyalg RSA -ext SAN=DNS:nastelin.com,IP:172.16.31.72

     • Create the client truststore by importing the CA certificate
       

       keytool -keystore kafka.client.truststore.jks -alias CARoot
       -importcert -file ca-cert.crt

     • Generate the certificate request for the client
       

       keytool -keystore kafka.client.keystore.jks -alias clientKeystore
       -certreq -file cert-file

     • Sign the certificate request with the CA
       

       openssl x509 -req -CA ca-cert.crt -CAkey ca-key.key -in cert-file
       -out cert-signed -days 365 -CAcreateserial -passin pass:meshiq
       -extfile san.conf -extensions req_ext

     • Import the CA certificate into the keystore
       

       keytool -keystore kafka.client.keystore.jks -alias CARoot
       -importcert -file ca-cert 

     • Import the signed certificate into the keystore
       

       keytool -keystore kafka.client.keystore.jks -alias clientKeystore
       -importcert -file cert-signed 

3. All the above created client certificates will appear as shown in the image below.
   
4. Ensure that the connectionsStanzaZK_KAFKA.cfg file has the correct configuration located at /opt/meshiq/kafka/tnt4j-stream-jmx/current/config
   
5. In tnt4j.KAFKA.properties located in /opt/meshiq/kafka/tnt4j-stream-jmx/current/config, make sure to provide the correct Host and Port.
   
6. stream-jmx-connect-file-config.sh file should be executed in order to collect the JMX metrics.
7. This file can be found under tnt4j-stream-jmx-VERSION-SNAPSHOT\run folder, also download the STREAM_JMX_SCRIPTS from the ftp site.
8. In order to execute the stream jmx config file firstly, add the following details to the stream-jmx-connect-file-config.sh file located in tnt4j-stream-jmx-VERSION-SNAPSHOT\run:
   

   TNT4JOPTS="$TNT4JOPTS
   TNT4JOPTS="$TNT4JOPTS
   -Djavax.net.ssl.keyStore=/opt/meshiq/kafka/Kafka-2.13-3.5.1/clientssl/kafka.client.keystore.jks /
   -Djavax.net.ssl.keyStorePassword=nastel /
   -Djavax.net.ssl.trustStore=/opt/meshiq/kafka/ Kafka-2.13-3.5.1/clientssl/kafka.client.truststore.jks /
   -Dcom.sun.management.jmxremote.rmi.port=9099 /
   -Dcom.sun.management.jmxremote.authenticate=false /
   -Dcom.sun.management.jmxremote.ssl=false"

9. Ensure that the following paths are correctly exported.
   
   Add the following lines as shown in the above screenshot.
   

   export TNT4J_PROPERTIES="$STREAM_JMX_HOME/../config/tnt4j.KAFKA.properties"
   
   export TNT4J_APPSERVER="KAFKA_TRACK"
   
   nohup $STREAM_JMX_HOME/stream-jmx-connect-file-config.sh ../config/connectionsStanzaZK_KAFKA.cfg&

10. Since the kafka server is configured with RMI Registry SSL, add the following properties as well: 
    

    ### --- Uncomment if server side has enabled RMI registry SSL communication
        CONN_OPTIONS="$CONN_OPTIONS -cp:com.sun.jndi.rmi.factory.socket=javax.rmi.ssl.SslRMIClientSocketFactory"
        ### ----------------------
    
    ###  --- Uncomment when following required if client has Java
        v11 and server has java v17   TNT4JOPTS="$TNT4JOPTS \ -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3\
        -Djdk.tls.namedGroups="secp256r1,secp384r1"\ -Dcom.sun.net.ssl.checkRevocation=false

11. After entering all the details correctly, you can start the stream using one of the following methods, depending on your setup:

    Option 1: Run the following shell command to start streaming data with the standard stream-jmx-connect-file-config.sh script:

    nohup $STREAM_JMX_HOME/stream-jmx-connect-file-config.sh ../config/connectionsStanzaZK_KAFKA.cfg &
    

    This method uses the standard streams package setup. Ensure that all paths, properties, and certificate references are configured correctly before running this command.

    Option 2: Alternatively, if you're using custom build startup shell scripts provided in the STREAM_JMX_SCRIPTS folder, run:

    ./KAFKA_STREAM_JMX_START.sh
    

    This script starts the stream using a custom configuration, typically used in deployments where additional startup logic or environment setup is handled within KAFKA_STREAM_JMX_START.sh.
    Use only one of the above methods based on how your environment is structured.

12. Once the script is executed, check whether JMX is running by using the command:
    

     ps -ef | grep jmx

    
    These lines depict that the JMX is enabled for Kafka running in the Kraft mode.
13. Kafka JMX with SSL metrics can now be monitored via the meshIQ platform.