This article explains how to configure Stream-JMX to collect JMX metrics from Apache Kafka brokers and forward them to meshIQ Observe.

Optionally, you can also configure Stream-JMX to collect ZooKeeper metrics if your Kafka deployment uses ZooKeeper instead of KRaft.

To collect operating system metrics, see Configure Collectd for OS Monitoring.

Prerequisites

Apache Kafka is installed and running.
Stream-JMX is installed.
Network connectivity to the Kafka JMX ports is available.

Procedure

Stream-JMX connects to Kafka brokers through JMX (Java Management Extensions), collects broker metrics, and forwards them to meshIQ Observe.

Install tnt4j-stream-jmx

Download the tnt4j-stream-jmx-<version>.tgz package and extract it to a directory of your choice.
```
tar -xzf tnt4j-stream-jmx-<version>.tgz
cd tnt4j-stream-jmx-<version>
```
The extracted directory contains:

Directory	Description
`bin/`	Shell scripts used to start tnt4j-stream-jmx.
`config/`	Configuration files that define Kafka connections and gateway settings which includes including `connections.cfg` and `tnt4j.properties`.
`lib/`, `opt/`	libraries used by stream-jmx.

Enable JMX on Kafka Brokers

Kafka brokers do not expose JMX by default. Configure a JMX port on each Kafka broker before starting the broker.

In the {KAKAFA_HOME}/bin directory, open the kafka-server-start.sh file and add the following line at the beginning of the file:

export JMX_PORT=9998

Save the file and restart Kafka for the changes to take effect.

You may choose any available port number. The only requirement is that it is not already in use on that host, otherwise you will get a port bind error.

Configure connections.cfg

Stream-JMX uses the connections.cfg file to determine which JVMs to monitor and how to collect JMX metrics.

Update the connections.cfg file located at config/connections.cfg directory.
Example configuration:

##############################################################################################################################################################
# VM endpoint configuration examples
##############################################################################################################################################################
# You can use variable expressions within "vm" or "vm.url" property definition. E.g.:
#
#      vm:                 service:jmx:rmi:///jndi/rmi://${vm.host}:${vm.port}/jmxrmi
#      vm.host:            192.0.2.10
#      vm.port:            9995;9996;9997
#
# will produce 3 VM definitions by combining defined single "vm.host" value and array of 3 "vm.port" values:
#
#      service:jmx:rmi:///jndi/rmi://192.0.2.10:9995/jmxrmi
#      service:jmx:rmi:///jndi/rmi://192.0.2.10:9996/jmxrmi
#      service:jmx:rmi:///jndi/rmi://192.0.2.10:9997/jmxrmi
#
# Yet if size of both "vm.host" and "vm.port" values are same, like this:
#
#      vm:                 service:jmx:rmi:///jndi/rmi://${vm.host}:${vm.port}/jmxrmi
#      vm.host:            192.0.2.10;192.0.2.11;192.0.2.12
#      vm.port:            9995;9996;9997
#
# then it will produce 3 VM definitions by mapping array values 1:1 as this:
#
#      service:jmx:rmi:///jndi/rmi://192.0.2.10:9995/jmxrmi
#      service:jmx:rmi:///jndi/rmi://192.0.2.11:9996/jmxrmi
#      service:jmx:rmi:///jndi/rmi://192.0.2.12:9997/jmxrmi
#
# Another way to get same result would be as this:
#
#      vm:                 service:jmx:rmi:///jndi/rmi://${vm.endpoint}/jmxrmi
#      vm.endpoint:         192.0.2.10:9995;192.0.2.11:9996;192.0.2.12:9997
#
##############################################################################################################################################################
# Kafka broker JMX connection definition - streams Kafka broker metrics to meshIQ.
##############################################################################################################################################################
{
   kafka.vm:                 service:jmx:rmi:///jndi/rmi://${vm.endpoint}/jmxrmi
   # CHANGE ME: broker host:port list (";"-separated for multiple brokers)
   kafka.vm.endpoint:        YOUR-BROKER-1-HOST:9998;YOUR-BROKER-2-HOST:9998;YOUR-BROKER-3-HOST:9998
   # CHANGE ME: Uncomment if your JMX endpoint requires authentication
   ; kafka.vm.user:           admin
   ; kafka.vm.pass:           admin
   kafka.vm.reconnect.sec:   10
   # agent.options: include filter ! exclude filter ! sample period ms ! initial delay ms ! batch size.
   # - Include kafka.*, org.apache.kafka.*, java.lang (JVM memory/GC/threading), java.nio (buffer pools).
   # - Exclude kafka.log:type=Log,name=LogStartOffset and LogEndOffset - noisy per-partition raw offset counters.
   # Note: exclude pattern needs a trailing ",*" since the real MBeans also carry topic=/partition= keys.
   kafka.agent.options:      kafka.*:*;\
                             org.apache.kafka.*:*;\
                             java.lang:*;\
                             java.nio:*\
                             !kafka.log:type=Log,name=LogStartOffset,*;\
                             kafka.log:type=Log,name=LogEndOffset,*\
                             !30000!5000!10
   kafka.agent.listener.flatMetricsMode:         true
   kafka.agent.listener.lowercasePropertyNames:  false
   kafka.agent.listener.excludedAttributes:      ClassPath,InputArguments,SystemProperties@java.lang:type=Runtime
   kafka.agent.listener.excludeOnError:          true
   # source.fqn is a free-form "#"-separated list of key=value pairs, later used for filtering.
   # A value is either dynamic (@bean:... - resolved live from the broker, like BROKER/HOST/CLUSTERID
   # below) or static/hardcoded (like ENV=DEV and ROLE=BROKER below).
   # CHANGE ME: add key=value pairs identifying where these metrics come from
   kafka.source.fqn:         BROKER=@bean:kafka.server:id=?,type=app-info#HOST=@sjmx.serverAddress#CLUSTERID=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value#ROLE=BROKER#ENV=DEV
}
##############################################################################################################################################################
# ZooKeeper JMX connection definition - streams ZooKeeper metrics to meshIQ.
##############################################################################################################################################################
# CHANGE ME: uncomment if not using KRaft.
; {
;     zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.endpoint}/jmxrmi
;     # CHANGE ME: ZK VM host:port (";"-separated for multiple ZK nodes)
;     zk.vm.endpoint:        YOUR-ZK-VM-HOST:9999
;     # CHANGE ME: Uncomment if your JMX endpoint requires authentication
; ;    zk.vm.user:            admin
; ;    zk.vm.pass:            admin
;     zk.vm.reconnect.sec:   10
;     # agent.options: include filter ! exclude filter ! sample period ms ! initial delay ms ! batch size.
;     # Include org.apache.ZooKeeperService (ZK's own metrics), java.lang, java.nio.
;     zk.agent.options:      org.apache.ZooKeeperService:*;\
;                            java.lang:*;\
;                            java.nio:*\
;                            !!30000!5000!10
;
;     zk.agent.listener.flatMetricsMode:         true
;     zk.agent.listener.lowercasePropertyNames:  false
;     zk.agent.listener.excludedAttributes:      ClassPath,InputArguments,SystemProperties@java.lang:type=Runtime
;     zk.agent.listener.excludeOnError:          true
;
;     # source.fqn is a free-form "#"-separated list of key=value pairs, later used for filtering.
;     # A value is either dynamic (@bean:... - resolved live from the ZK node, like SERVER/HOST
;     # below) or static/hardcoded (like ROLE=ZOOKEEPER and ENV=DEV below).
;     # CHANGE ME: add key=value pairs identifying where these metrics come from
;     zk.source.fqn:         SERVER=@bean:org.apache.ZooKeeperService:name0=?#HOST=@sjmx.serverAddress#ROLE=ZOOKEEPER#ENV=DEV
; }

The token kafka in each property name is the connection group identifier. You can rename it, as long as it stays consistent across the block.

Understanding the Configuration Properties

vm

Defines the Java Management Extensions (JMX) connection URL template.

The vm.endpoint property, described below, supplies the host and port values used by this URI.
vm.endpoint

Specifies the Kafka broker JMX endpoints in the format host:port. Use a semicolon (;) to specify multiple brokers (for example, broker1:9998;broker2:9998).

Update this property to match the JMX connection endpoints for the Kafka brokers in your environment.
vm.user and vm.pass

Specify the credentials required for JMX authentication.

In most environments, JMX authentication is not enabled, so these properties remain commented out. Enable and configure them only if your JMX connections require authentication.
vm.reconnect.sec

Specifies how often stream-jmx attempts to reconnect after a failed connection. For example, 10 means it retries every 10 seconds.
agent.options

The agent.options property controls which JMX MBeans are monitored and how frequently data is collected. Configuring this property appropriately helps reduce unnecessary data collection and optimizes storage.

This is a composite property where individual settings are separated by the ! character. The sample configuration uses line continuation (\) characters to split the property across multiple lines for readability, but it is treated as a single property.
- Token 1 – Included MBeans
  
  Specifies the MBeans to monitor using JMX ObjectName patterns. You can monitor everything (*:*), an entire domain (for example, kafka.*), or specific MBeans.
  
  The sample configuration monitors only kafka.*, org.apache.kafka.*, java.lang (JVM memory, garbage collection, and threads), and java.nio (buffer pools) to collect only the metrics required for Kafka monitoring.

Token 2 – Excluded MBeans

Specifies MBeans to exclude from monitoring. This is useful for omitting noisy or low-value metrics. The sample configuration excludes the LogStartOffset and LogEndOffset MBeans because they generate large amounts of per-partition data that is typically not required for monitoring.
Token 3 – Sampling Interval

Specifies how often MBean data is collected, in milliseconds.

Example value: 30000

This value causes Stream-JMX to collect metrics every 30 seconds.

Increase or decrease this value to control the sampling frequency.
Token 4 – Sampling Startup Delay

Specifies the delay, in milliseconds, before metric collection begins after the application starts.

Default value: 5000

This delays metric collection for 5 seconds, allowing the application to complete its initialization before sampling begins.

In most cases, this value should not be changed.
Token 5 – Batch Size

Specifies the number of sampled MBeans to accumulate before sending data to Observe.

Example value: 10

With this setting, Stream-JMX sends data after every 10 MBeans are collected instead of waiting for the entire sampling cycle to complete.

In most environments, this value should not be changed.

When specifying an MBean pattern that has additional key properties, append ,* to the ObjectName pattern. For example, use kafka.log:type=Log,name=LogEndOffset,* instead of kafka.log:type=Log,name=LogEndOffset. Without the trailing ,*, the pattern requires an exact match and will not match MBeans that also include additional properties such as topic and partition.

agent.listener.*

Controls how collected metrics are formatted before being sent to Observe (for example, metric naming and attribute filtering). The default settings such as flatMetricsMode, lowercasePropertyNames, excludedAttributes, excludeOnError are recommended and usually do not require modification.
source.fqn

Defines the Fully Qualified Name (FQN) that uniquely identifies the JMX data source in Observe.

The property consists of a series of key=value pairs separated by the # character. These values create a hierarchy that helps group and identify monitored resources.

Observe uses the Source FQN to organize monitored resources into a hierarchical Resource Groups. The hierarchy is evaluated from right to left, where the rightmost element represents the highest level.

Source Hierarchy example:

RootFQN
└── ENV=DEV
  └── ROLE=BROKER
      └── CLUSTERID=
          └── HOST=
              └── BROKER=

The Source FQN defined in this file is combined with the value of the source.factory.RootFQN property in the tnt4j.properties file.

The resulting hierarchy is:

Level	Description
`RootFQN`	The top-level grouping defined by `source.factory.RootFQN` in `tnt4j.properties`.
`ENV=DEV`	Identifies the environment where the broker is running. Update this value to match your environment, such as `DEV`, `TEST`, or `PROD`.
`ROLE=BROKER`	Identifies the type of monitored component. This value distinguishes Kafka brokers from other monitored services such as ZooKeeper or Schema Registry.
`CLUSTERID=`	Dynamically retrieved from Kafka JMX and used to identify the Kafka cluster.
`HOST=`	Dynamically populated with the JMX connection IP address. If host names are preferred, use `@sjmx.serverName` instead of `@sjmx.serverAddress`.
`BROKER=`	Dynamically retrieved from Kafka JMX and used to identify the individual broker within the cluster.

This hierarchy uniquely identifies each Kafka broker monitored by Stream-JMX within Observe.

Optional: Configure ZooKeeper Monitoring

If your Kafka deployment uses ZooKeeper, uncomment the ZooKeeper section in connections.cfg and update the following properties:

ZooKeeper JMX endpoint
Authentication credentials (if required)
Source FQN values

No additional configuration is required for KRaft-based Kafka clusters.

Configure tnt4j.properties

The tnt4j.properties, located at config/tnt4j.properties defines how the stream-jmx sends data to the meshIQ Data Services Gateway.

Update the tnt4j.properties file with the required configuration changes described below, and then save the file:

The following example shows the default Stream-JMX TNT4J sink configuration:

;Default tracking configuration for all sources (source: *), used only if no other stanza matches.
{
    source: *
    source.factory: com.jkoolcloud.tnt4j.source.SourceFactoryImpl
    source.factory.RootFQN: RUNTIME=?#SERVER=?#NETADDR=?
    tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
    dump.sink.factory: com.jkoolcloud.tnt4j.dump.DefaultDumpSinkFactory
    event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory
    event.formatter: com.jkoolcloud.tnt4j.format.SimpleFormatter
    tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
    tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}
;Stanza used for Stream-JMX sources
{
    source: com.jkoolcloud.tnt4j.stream.jmx
    source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
    ; CHANGE ME (optional): GENERIC=JMX sets the category name. Use something more specific if you like, e.g. GENERIC=KAFKA-JMX
    source.factory.RootFQN: GENERIC=JMX
    source.factory.RootSSN: tnt4j-stream-jmx
    tracker.factory: com.jkoolcloud.tnt4j.tracker.DefaultTrackerFactory
    event.sink.factory: com.jkoolcloud.jesl.tnt4j.sink.JKCloudEventSinkFactory
    ; CHANGE ME: meshIQ Data Services Gateway URL
    event.sink.factory.Url: http://YOUR-GATEWAY-HOST:6580/
    ; CHANGE ME: your streaming token
    event.sink.factory.Token: YOUR-STREAMING-TOKEN
    event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
    ; Normalize characters that are inconvenient in a metric property name.
    event.formatter.KeyReplacements: " "->"_" "\""->"_" "'"->"_" "\\\\"->"_"
    tracking.selector: com.jkoolcloud.tnt4j.selector.DefaultTrackingSelector
    tracking.selector.Repository: com.jkoolcloud.tnt4j.repository.FileTokenRepository
}

Update the following properties in tnt4j.properties file:

Property	Required Change
`event.sink.factory.Url`	Replace `<gateway-host>` with the hostname or IP address of your meshIQ Data Services Gateway.
`event.sink.factory.Token`	Replace `<streaming-token>` with your streaming token.
`source.factory.RootFQN`	This property defines the resource hierarchy in Observe. Ensure that the reserved `GENERIC` token is the rightmost token in this property, as its value determines the category under which Kafka resources are displayed in the Observe Resource Selector.

The GENERIC token must be defined only in the source.factory.RootFQN property in the tnt4j.properties file. Do not define the GENERIC token in the kafka.source.fqn property in the connections.cfg file.

The event.formatter.KeyReplacements property is preconfigured with the recommended default value and typically does not require any changes.

Start Stream-JMX

To start Stream-JMX for Kafka monitoring, run the following command from {JMX_HOME}/bin location.

### Define additional JVM System properties used by stream-jmx
#export TNT4JOPTS="-Dtnt4j.stream.jmx.sampler.mbeans.query.retry.max.attempts.count=0"
 ./stream-jmx-connect-file-config.sh ../config/connections.cfg

The optional export TNT4JOPTS environment variable specifies additional JVM system properties used by Stream-JMX.
./bin/stream-jmx-connect-file-config.sh starts the Stream-JMX service.
./config/connections.cfg specifies the path to the Stream-JMX connection configuration file.

Verify Metric Collection

After Stream-JMX starts, verify that the connection was established successfully.

Check the Stream-JMX log for an entry beginning with SamplingAgent.startSampler:. This confirms that the connection to the Kafka brokers was established successfully and source.fqn was built correctly.
Verify that periodic log entries beginning with DefaultSampleListener - Post: are being generated. These entries indicate that Stream-JMX is collecting and sending metrics.
Confirm that total.error.count=0 appears in the log. A value of 0 indicates that metrics are being collected and transmitted without errors.

Verify Metrics in Observe

After Stream-JMX starts successfully, it begins collecting JMX metrics from the configured Kafka brokers and sends them to meshIQ Observe.

Sign in to the meshIQ Observe web interface.
Create a Resource Group.
Verify that the value assigned to GENERIC in the source.factory.RootFQN property of tnt4j.properties is listed as a category in the Category drop-down list of the Observe, alongside the existing resource categories. ( for example: JMX)
After selecting the category, the available JMX MBean domains are displayed as Resource Types.
The remaining elements of the source.fqn property define the resource hierarchy displayed in Observe.

These values are automatically populated from the JMX MBeans referenced in the source.fqn configuration in connections.cfg file.
Create a Monitor. All metrics collected for the selected resource are available for alert configuration.

Metric names are generated from the sampled JMX MBean attributes. When flatMetricsMode=true, the MBean type , name , and attribute are combined into a single metric name, such as Partition.ReplicasCount.Value.
Configure alert thresholds for the required metrics and save the Monitor.

For additional information about tnt4j-streams-jmx, see docs/tnt4j-stream-jmx.md in the extracted package.

Configure Stream-JMX for Kafka Monitoring in meshIQ Observe

Prerequisites

Procedure

Install tnt4j-stream-jmx

Enable JMX on Kafka Brokers

Configure connections.cfg

Understanding the Configuration Properties

`vm`

`vm.endpoint`

`vm.user` and `vm.pass`

`vm.reconnect.sec`

`agent.options`

Token 1 – Included MBeans

Token 2 – Excluded MBeans

Token 3 – Sampling Interval

Token 4 – Sampling Startup Delay

Token 5 – Batch Size

`agent.listener.*`

`source.fqn`

Optional: Configure ZooKeeper Monitoring

Configure tnt4j.properties

Start Stream-JMX

Verify Metric Collection

Verify Metrics in Observe

Prerequisites

Procedure

Install tnt4j-stream-jmx

Enable JMX on Kafka Brokers

Configure connections.cfg

Understanding the Configuration Properties

vm

vm.endpoint

vm.user and vm.pass

vm.reconnect.sec

agent.options

Token 1 – Included MBeans

Token 2 – Excluded MBeans

Token 3 – Sampling Interval

Token 4 – Sampling Startup Delay

Token 5 – Batch Size

agent.listener.*

source.fqn

Optional: Configure ZooKeeper Monitoring

Configure tnt4j.properties

Start Stream-JMX

Verify Metric Collection

Verify Metrics in Observe

`vm`

`vm.endpoint`

`vm.user` and `vm.pass`

`vm.reconnect.sec`

`agent.options`

`agent.listener.*`

`source.fqn`