How do I manually update the Kafka component?

This article explains how to manually update the Kafka component used by the meshIQ Platform.

Starting with meshIQ Platform v11.3.2, Kafka is included as part of the platform packages. We recommend using the Kafka version provided with your platform release because it has been validated for compatibility.

For releases earlier than v11.3.2, contact meshIQ Support to determine the appropriate Kafka version.

Before updating Kafka:

• Back up the existing Kafka installation and configuration.
• Ensure you have administrator/root access.
• Stop all Track services that connect to Kafka.

Update the Kafka package

1. Stop the Kafka service by using the command ./stop.sh kafkasrv in $APIN_HOME/sbin.
2. Navigate back to Kafka directory present in $APIN_HOME/kafka.

3. Untar the downloaded Kafka package using the command. 

   tar -zxvf kafka2.13-xxx.tar.gz

4. Kafka2.13-xxx folder is obtained which will be the current version.

5. Unlink the old version of Kafka by using the command.

   Unlink Current

6. Link the current version of kafka by using the command. 

   ln -s kafka2.13-xxx current

7. Open the untared folder and edit the server.properties file present in the $APIN_HOME/kafka/kafka2.13-xxx/config/.
    

   cd/$APIN_HOME/kafka/kafka2.13-xxx/config/
   view server.properties

8. Add the following properties under Server Basics:

   message.max.bytes=10485760
   replica.fetch.max.bytes=10485760

9. Under zookeeper add the following line

   zookeeper.connect=localhost:2181/xraykafka

10. Change the following properties: 

    • Under Socket Server Settings:

      Change num.network.threads from 3 to 8

      num.network.threads=8

      Change num.io.threads from 8 to 16
      

      num.io.threads=16

    • By default, log.dirs is set to the folder shown below. But you can change it if you do not want to use the default folder.
      

      log.dirs=/tmp/kafka-logs

    • Under Log Basics:
      Change num.partitions from 1 to 16
      

      num.partitions=16

    • Under Log Flush Policy, uncomment the following line:
      

      log.segment.bytes=107374182

11. Save the server.properties file after the above changes.
12. After completion of all steps, restart the Kafka service from /opt/meshiq/sbin by using the command ./start.sh kafkasrv to apply the changes.
13. Recreate Kafka topics using the command ./config.sh kaf in $APIN_HOME/sbin.