Using Stream-JMX

This article provides instructions for running and configuring Stream-JMX to monitor JVMs locally or remotely using different connection methods.

Using Stream-JMX

To use Stream-JMX, do one of the following:

• run Stream-JMX as a -javaagent
• attach Stream-JMX as an agent to a running JVM
• connect Stream-JMX over JMXConnector to a locally running JVM or a remote JMX service

• embed Stream-JMX code into your application by using this Maven dependency:

      <dependency>
          <groupId>com.jkoolcloud.tnt4j.stream</groupId>
          <artifactId>tnt4j-stream-jmx-core</artifactId>
          <version>12.0.5</version>
      </dependency>

Running Stream-JMX as -javaagent, attaching it to a running JVM, or connecting over JMXConnector to locally running JVM or remote JMX service over RMI connection can be invoked without changing your application code.

When you want to sample JMX inter-process communication way (providing -connect or -attach mode -vm parameter value as PID/process name), then Stream-JMX runner and sampled process VMs MUST be:

• built by same vendor (e.g. IBM, Oracle, OpenJDK)
• of same architecture (x86, x64)

• Stream-JMX runner VM version must be higher or equal compared to the sampled VM's version - e.g. trying to sample JVM 10+ using JVM 8 may produce exceptions like this:

  java.io.IOException: Non-numeric value found - int expected
    at sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:299)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:63)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgentLibrary(HotSpotVirtualMachine.java:79)
    at sun.tools.attach.HotSpotVirtualMachine.loadAgent(HotSpotVirtualMachine.java:103)
    at com.sun.tools.attach.VirtualMachine.loadAgent(VirtualMachine.java:540)

• Check if JDK attach tooling is enabled and bound:
  • For Java versions prior to 9, check which JDK tools.jar is referred over LIBPATH variable, it also must match runner JVM

  • For Java versions 9+, check if jdk.attach module is enabled over --add-opens and --add-exports arguments:

    java --add-exports=jdk.attach/sun.tools.attach=ALL-UNNAMED --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED ...

If JVM vendors, architecture, or versions do not match conditions above - then only way to collect JMX samples is by using -connect mode defining JMXConnector (over RMI) URL.

When running stream-jmx sometimes some Unsupported type exceptions may appear in the console/log, e.g.:

Failed to sample:
 ojbName=java.lang:type=MemoryPool,name=PS Eden Space,
 info=javax.management.MBeanAttributeInfo[description=UsageThresholdExceeded, name=UsageThresholdExceeded, type=boolean, read-only, isIs, descriptor={openType=javax.management.openmbean.SimpleType(name=java.lang.Boolean), originalType=boolean}],
 exclude=false,
 ex=javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
javax.management.RuntimeMBeanException: java.lang.UnsupportedOperationException: Usage threshold is not supported
    ... 25 more
Caused by: java.lang.UnsupportedOperationException: Usage threshold is not supported
    ... 27 more

It is OK, since some MBeans do not provide some attributes depending on Operating System or other run environment setup. If you run JConsole you will get values marked as Unavailable for such attributes painted in red (e.g. see java.lang -> MemoryPool -> PS Eden Space attribute UsageThresholdExceeded). These Unsupported exceptions are shown on first iteration of sampling for the user to know that all MBean attributes were accessed.

Argument definitions:

• -vm: - virtual machine descriptor. It can be PID or JVM process name fragment. * value is wildcard to pick all found VMs, running on local machine.
• -ao: - agent options string using ! symbol as delimiter. Options format: mbean-filter!exclude-filter!sample-ms!init-delay-ms!batch-size
  • mbean-filter - MBean include name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter (domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3) or grouping using () and | notation. Grouping like java.lang:type=(Threading|Memory|OperatingSystem|Runtime) is equal to java.lang:type=Threading;java.lang:type=Memory;java.lang:type=OperatingSystem;java.lang:type=Runtime;. Default is *:*.
  • exclude-filter - MBean exclude name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter (domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3) or grouping using () and | notation. Grouping like java.lang:type=(Threading|Memory|OperatingSystem|Runtime) is equal to java.lang:type=Threading;java.lang:type=Memory;java.lang:type=OperatingSystem;java.lang:type=Runtime;. Optional, default is ``.
  • sample-ms - MBeans sampling rate in milliseconds. Optional, Default is 30 sec. NOTE: if MBean sampling duration gets longer than configured, it will dynamically reschedule sampling from fixed rate to fixed delay in sample-ms between samples.
  • init-delay-ms - MBeans sampling initial delay in milliseconds. Optional, by default it is equal to sample-ms value.
  • batch-size - number of sampled MBeans to post over single package. -1 means unlimited batch - all MBeans from sampling iteration are packed into single package. Optional, default is 20. NOTE: if current MBean sampling iteration takes longer than system property tnt4j.stream.jmx.sampler.batch.period.sec defined value or used memory consumption percentage gets higher than system property tnt4j.stream.jmx.sampler.batch.used.memory.percent defined value, current batch is also considered to be complete.
• -cp: - JMX connection parameter string using = symbol as delimiter. Defines only one parameter, to define more than one use this argument multiple times. Argument format: -cp:key=value. See Java SE monitoring and management documentation for more details.
• -slp: - sampler parameter string using = symbol as delimiter. Defines only one parameter, to define more than one use this argument multiple times. Argument format: -slp:key=value
  • trace - flag indicating whether the sample listener should print trace entries to print stream. Default value - false.
  • forceObjectName - flag indicating to forcibly add objectName attribute if such is not present for a MBean. Default value - false.
  • addStatisticMetadata - flag indicating to add J2EE metadata entries as attributes for a MBean. Default value - false.
  • compositeDelimiter - delimiter used to tokenize composite/tabular type MBean properties keys. Default value - \.
  • useObjectNameProperties - flag indicating to copy MBean ObjectName contained properties into sample snapshot properties. Default value - true.
  • excludeOnError - flag indicating to auto-exclude failed to sample attributes. Default value - false.
  • excludedAttributes - list of user chosen attribute names (may have wildcards * and ?) to exclude, pattern: attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName.
• -sp: - sampler JVM system property string using = symbol as delimiter. Defines only one system property, to define more than one use this argument multiple times. Argument format: -sp:key=value. NOTE: This argument allows to use su.sh or su.cmd encoded property values (e.g. passwords).
• -ssl - flag indicating to disable SSL verification in case SSL is misconfigured: expired, self-signed, or domain mismatching certificate.

Executable OS shell run script files bin/stream-jmx.bat or bin/stream-jmx.sh are dedicated for this task.

Command line to run stream-jmx as JVM agent looks like this:

java -javaagent:tnt4j-stream-jmx-core-all.jar="*:*!30000" -Dtnt4j.config=tnt4j.properties -Dlog4j2.configurationFile=log4j2.xml -classpath "tnt4j-stream-jmx.jar;lib/*" your.class.name your-args

In case your Java runtime is version 9+, add these Java command line arguments:

--add-exports=java.base/jdk.internal.loader=ALL-UNNAMED --add-opens=java.base/jdk.internal.loader=ALL-UNNAMED

Executable OS shell run script files bin/stream-jmx-attach.bat or bin/stream-jmx-attach.sh are dedicated to do the job.

Command line to attach to a local JVM process JMX looks like this:

java -Dtnt4j.config=./config/tnt4j.properties -Dlog4j2.configurationFile=./config/log4j2.xml -classpath "tnt4j-stream-jmx-core-all.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -attach -vm:activemq -ap:tnt4j-stream-jmx-core-all.jar -ao:*:*!10000

SamplingAgent arguments -attach -vm:activemq -ap:tnt4j-stream-jmx-core-all.jar -ao:*:*!10000 states:

• -attach - defines that SamplingAgent shall be attached to running JVM process
• -vm:activemq - is JVM descriptor. In this case it is running JVM name fragment activemq. But it also may be JVM process identifier - PID. Mandatory argument.
• -ap:tnt4j-stream-jmx-core-all.jar - is agent library name. If it is class path - then only name should be sufficient. In any other case define full or relative path, e.g., ../build/tnt4j-stream-jmx/tnt4j-stream-jmx-12.0.5/tnt4j-stream-jmx-core-all.jar. Mandatory argument.
• -ao:*:*!10000 - is JMX sampler options stating to include all MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
• -slp: - any JMX sampler configuration property. See Program arguments used for details.
• -sp: - any JVM system property used by sampler. See System properties used for details. This argument allows to use su.sh or su.cmd encoded property values (e.g. passwords).

Arguments and properties defined running SamplingAgent.main is forwarded to SamplingAgent agent attached to JVM process.

If you get exception com.sun.tools.attach.AgentInitializationException: Agent JAR loaded but agent failed to initialize while running stream-jmx in -attach mode it is most likely that some stream-jmx used classes are not resolved in attach JVM class path. Pay attention to use tnt4j-stream-jmx-core-[VERSION]-all.jar as agent lib to have all dependent classes available within one jar. If you are trying to attach application implementing J2EE API, use tnt4j-stream-jmx-j2ee-[VERSION]-all.jar. Same approach should be followed when attaching WAS and Liberty instances - use -all.jar for matching product API.

You can attach SamplingAgent to JVM from your custom API by calling SamplingAgent.attach(String,String,String) method.

Sample attach call:

try {
  SamplingAgent.attach("activemq", "tnt4j-stream-jmx-core-all.jar", "*:*!10000");
} catch (Exception exc) {
  exc.printStackTrace();

Executable OS shell run script files bin/stream-jmx-connect.bat or bin/stream-jmx-connect.sh are dedicated to do the job:

stream-jmx-connect script has 4 parameters:

1. process id or service URI (required)
2. agent options for MBeans include/exclude filter and sampling interval (optional, . value sets default)
3. service identifier for the process/service being monitored (optional, . value sets default)
4. sampling agent arguments (optional, . value sets default)

MS Windows

:: using URL
/bin/stream-jmx-connect.bat service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi
:: using URL with connection parameters
/bin/stream-jmx-connect.bat service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi . . -ul:admin -up:admin -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory
:: using process name part
/bin/stream-jmx-connect.bat activemq
:: using pid
/bin/stream-jmx-connect.bat 1553 

*nix

# using URL
./bin/stream-jmx-connect.sh "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi"
# using URL with connection parameters
./bin/stream-jmx-connect.sh "service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi" . . "-ul:admin -up:admin -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory"
# using process name part
./bin/stream-jmx-connect.sh activemq
# using pid
./bin/stream-jmx-connect.sh 1553

Command line to connect to a local JVM process JMX looks like this:

java -Dtnt4j.config=./config/tnt4j.properties -Dlog4j2.configurationFile=./config/log4j2.xml -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -connect -vm:activemq -ao:*:*!*:dummy!10000

SamplingAgent arguments -connect -vm:activemq -ao:*:*!*:dummy!10000 states:

• -connect - defines that SamplingAgent shall connect to running JVM process over JMXConnector (RMI) connection.
• -vm:activemq - is JVM descriptor. In this case it is running JVM name fragment activemq. But it also may be JVM process identifier - PID. Mandatory argument.
• -ao:*:*!*:dummy!10000 - is JMX sampler options stating to include all MBeans, exclude all dummy MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
• -slp: - any JMX sampler configuration property. See Program arguments used for details.
• -sp: - any JVM system property used by sampler. See System properties used for details. NOTE: This argument allows to use su.sh or su.cmd encoded property values (e.g. passwords).

Command line to connect remote JMX service looks like this:

java -Dtnt4j.config=./config/tnt4j.properties -Dlog4j2.configurationFile=./config/log4j2.xml -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -connect -vm:service:jmx:<JMX_URL> -ul:admin -up:admin -ao:*:*!!10000 -cri:30 -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory

SamplingAgent arguments -connect -vm:service:jmx:<JMX_URL> -ul:admin -up:admin -ao:*:*!!10000 -cri:30 -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory states:

• -connect - defines that SamplingAgent shall connect to running JMX service over JMXConnector (RMI) connection.
• -vm:service:jmx:<JMX_URL> - is JMX service URL to use for connection. Mandatory argument. Full URL may be like service:jmx:rmi://127.0.0.1/stub/rO0ABXN9AAAAAQAlamF2YXgubWFuYWdlbWVudC5yZW1vdGUucm1pLlJNSVNlcnZlcnhyABdqYXZhLmxhbmcucmVmbGVjdC5Qcm94eeEn2iDMEEPLAgABTAABaHQAJUxqYXZhL2xhbmcvcmVmbGVjdC9JbnZvY2F0aW9uSGFuZGxlcjt4cHNyAC1qYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN0SW52b2NhdGlvbkhhbmRsZXIAAAAAAAAAAgIAAHhyABxqYXZhLnJtaS5zZXJ2ZXIuUmVtb3RlT2JqZWN002G0kQxhMx4DAAB4cHc2AAtVbmljYXN0UmVmMgAACzE3Mi4xNi42Ljg2AADPWKO5DJD/bZIhG9aBuwAAAVo8DdAkgAEAeA== .
• -ul:admin - is user login. In this case it is admin. User login argument is optional.
• -up:admin - is user password. In this case it is admin. User password argument is optional. 
  NOTE: value for this argument can be encoded using su.sh or su.cmd.
• -ao:*:*!!10000 - is JMX sampler options stating to include all MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
• -cp:java.naming.security.authentication=simple -cp:java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory - is JMX connector parameters definitions in properties format key=value. JMX connector parameters are optional and can be defined multiple times - as many as there are required JMX connector parameters. See Java API Context class documentation for available properties naming. NOTE: If you are using some API extending JNDI, check documentation if it provides some additional connection configuration properties.
• -cri:30 - is connection retry interval in seconds. In this case it is 30 sec. between connect retry attempts. Connection retry interval is optional - default value is 10 sec.. Special values are:
  • 0 indicates no delay between repeating connect attempts.
  • -1 indicates no repeating connect attempts shall be made at all and application has to stop on first failed attempt to connect.
• -slp: - any JMX sampler configuration property. See Program arguments used for details.
• -sp: - any JVM system property used by sampler. See System properties used for details. NOTE: This argument allows to use su.sh or su.cmd encoded property values (e.g. passwords).

NOTE:

• URI of remote RMI service (e.g., to connect remote Kafka service) may require additional / chars:

  -vm:service:jmx:rmi:///jndi/rmi:///[HOST]:[PORT]/jmxrmi

  e.g.:

  -vm:service:jmx:rmi:///jndi/rmi:///172.16.6.35:2181/jmxrmi

• Remote end may require to enable JMX listening port in service configuration to accept connections (e.g., to connect remote Kafka service):

  export JMX_PORT=${JMX_PORT:-9999}

• Tomcat and Kafka do not provide J2EE implementation; therefore, you only need the stream-jmx core jar in the classpath when sampling Tomcat/Kafka metrics over JMX. Executable OS shell run script files use only core as the MODULE_SET variable value:

  • bin/stream-jmx-connect.bat

    set "MODULE_SET=core"

  • bin/stream-jmx-connect.sh

    MODULE_SET=("core")

• See Enabling Tomcat JMX Remote to enable remote JMX access of Tomcat server instance. If your Tomcat server is behind firewall - also refer to JMX Remote Lifecycle Listener on how to set up such Tomcat server instance to be accessible from outside.
• See ActiveMQ JMX how to enable remote JMX access for ActiveMQ Classic.
• See Artemis Management how to enable remote JMX access for ActiveMQ Artemis.

If sampling agent finds multiple VMs matching provided JVM descriptor, it will bind and sample JMX beans from all these VM's, e.g.:

./bin/stream-jmx-connect.sh kafka

Dedicated SamplingAgent arguments:

• -f: - defines path for external configuration file, containing set of VM connection definitions.

To run JMX samples streaming for multiple VMs define in external configuration file use stream-jmx-connect-file-config.bat (.sh) files, e.g.: *nix

./bin/stream-jmx-connect-file-config.sh ./config/connections.cfg

MS Windows

\bin\stream-jmx-connect-file-config.bat \config\connections.cfg

When connections configuration gets too complex to be defined using tokenized lines, there is possibility to define connections using properties divided into stanzas. One stanza defines one VM descriptor (connection) properties, like this:

##############################################################################################################################################################
{
    zk.vm:                  service:jmx:rmi:///jndi/rmi://localhost:9995/jmxrmi
    zk.vm.user:             admin
    zk.vm.pass:             admin
    zk.vm.reconnect.sec:    10
    zk.agent.options:       java.lang:*!!60000
    zk.source.fqn:          SERVICE=@bean:org.apache.ZooKeeperService:name0=*/?ClientPort#SERVER=@bean:java.lang:type=Runtime/?Name
}
##############################################################################################################################################################
{
    kafka.vm:               kafka:zk://127.0.0.1:2181
    kafka.vm.user:          admin
    kafka.vm.pass:          admin
    kafka.vm.reconnect.sec: 10
    kafka.agent.options:    java.lang:*!!60000
    kafka.source.fqn:       SERVICE=@bean:java.lang:type=Runtime/?Name#SERVER=@bean:kafka.server:id=?,type=app-info#DATACENTER=@bean:kafka.server:type=KafkaServer,name=ClusterId/?Value
}
##############################################################################################################################################################
{
    solr.vm:                 solr:zk://172.16.6.208:2181
    solr.vm.user:            admin
    solr.vm.pass:            admin
    solr.vm.reconnect.sec:   10
    solr.agent.options:      java.lang:*!!5000
    solr.source.fqn:         SERVER=@bean:solr:dom1=core,dom2=?,dom3=*,reporter=*,category=*,scope=core,name=*
    solr.custom.urlPattern:  service:jmx:rmi:///jndi/rmi://{0}:{1,number,######}/jmxrmi
    solr.custom.port:        18983
}
##############################################################################################################################################################

• vm or vm.url - defines VM or (VM coordinated instance) URL.
• vm.user - defines VM connection used user-name.
• vm.pass or vm.password - defines VM connection used user password. NOTE: value for this argument can be encoded using su.sh or su.cmd.
• vm.reconnect.sec - defines period in seconds to reconnect if VM connection gets interrupted. Negative value means do not try to reconnect. Default value - 10 sec..
• vm.reconnect.max.count - defines number of retries to connect if VM connection gets interrupted. Negative value means do not try to reconnect. Default value - infinitely.
• agent.options - defines agent sampling options.
• source.fqn - defines TNT4J Source FQN values mapping used by VM descriptor.
• custom - defines group of additional custom VM descriptor properties:
  • urlPattern - defines JMX service URL pattern to be filled in by ZooKeeper/KRaft coordinated VMs resolver. Default value - service:jmx:rmi:///jndi/rmi://{0}:{1,number,######}/jmxrmi, where {0} is replaced with JMX service runner machine host name/IP and {1} is replaced with JMX service runner machine port number.
  • port - defines JMX service port number to be used by ZooKeeper/KRaft coordinated VMs resolver. Some services (like Apache Solr or Kafka KRaft) does not directly provide JMX service port, so it is needed to define it manually. Apache Kafka for instance - does provide JMX connection port, so there is no need to define it additionally - it is picked automatically from ZooKeeper node provided data.
  • brokers.refresh.period.sec - defines KRaft VM resolver used brokers refresh period defined in seconds. Default value - 30 sec..
  • broker.port.property - defines KRaft VM resolver used broker configuration property providing VM JMX port value. NOTE: at the moment does nothing, because Kafka hides values for custom properties when accessed over Kafka Admin Client. Reserved for the future.
• zk - defines group of ZooKeeper client configuration properties, see Client Configuration
• admin - defines group of Kafka Admin client configuration properties, see Admin Configs

You can use variable expressions within vm or vm.url property definition. Array values are delimited using ; symbol. For example:

zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.host}:${vm.port}/jmxrmi
zk.vm.host:            172.16.6.208
zk.vm.port:            9995;9996;9997

will produce 3 VM definitions by combining defined single vm.host value and array of 3 vm.port values:

service:jmx:rmi:///jndi/rmi://172.16.6.208:9995/jmxrmi
service:jmx:rmi:///jndi/rmi://172.16.6.208:9996/jmxrmi
service:jmx:rmi:///jndi/rmi://172.16.6.208:9997/jmxrmi

If size of both vm.host and vm.port values are same, like below:

zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.host}:${vm.port}/jmxrmi
zk.vm.host:            172.16.6.208;172.16.6.210;172.16.6.212
zk.vm.port:            9995;9996;9997

service:jmx:rmi:///jndi/rmi://172.16.6.208:9995/jmxrmi
service:jmx:rmi:///jndi/rmi://172.16.6.210:9996/jmxrmi
service:jmx:rmi:///jndi/rmi://172.16.6.212:9997/jmxrmi

Another way to get same result would be as:

zk.vm:                 service:jmx:rmi:///jndi/rmi://${vm.address}/jmxrmi
zk.vm.address:         172.16.6.208:9995;172.16.6.210:9996;172.16.6.212:9997

See TNT4J Source fields configuration as base Source fields configuration reference.

Multiple VMs monitoring requires additional data Source configuration when you want to distinguish where JMX data came from.

When all monitored VM's can uniquely identify itself using same SourceFQN by using dynamic MBean property value (common when monitoring same service on different machines), e.g.

source.factory.SERVICE: @bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
source.factory.SERVER: @bean:java.lang:type=Runtime/?Name
source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?

Then no additional source configuration is required.

But when your services ecosystem is widely distributed running different services on different machines (or multiple different services on same machine), every service identifies itself over different MBean property. In this case you can configure source RootFQN in tnt4j.properties and define additional SourceFQN path fragment for every monitored VM. This way final VM MBeans sample will have SourceFQN made by Stream-JMX by concatenating tnt4j.properties defined RootFQN with that additional VM SourceFQN path fragment.

For example, when tnt4j.properties defines RootFQN as:

source.factory.GENERIC: Streams
source.factory.DATACENTER: HQDC
source.factory.SERVER: @bean:java.lang:type=Runtime/?Name
source.factory.RootFQN: SERVER=?#DATACENTER=?#GENERIC=?

and having VMs configured this way:

SERVICE=@bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
SERVICE=@bean:kafka.server:id=?,type=app-info 

• localhost - `SERVICE=ID:PC-NAME-52295-1538060220413-0:1#SERVER=2453@PC-NAME#DATACENTER=HQDC
• 192.168.1.1 - `SERVICE=3SERVER=2457@boxName#DATACENTER=HQDC

NOTE: VM additional SourceFQN path must be defined following the same rules as in TNT4J config:

SourceType1=value1#SourceType2=value2#...#SourceTypeN=valueN

Stream-JMX has the ability to resolve JMX service connections for VMs coordinated by ZooKeeper or Apache KRaft. In this case you'll need:

1. Use tnt4j-stream-jmx-zk or tnt4j-stream-jmx-kraft module dependency (for development) or use built jar of this module in your JVM classpath (for deployment). Also see bin/stream-jmx-connect.bat (.sh) files for commented out variable MODULE_SET definition combinations.
2. Use predefined VM descriptor prefixes to initiate VMs JMX connection resolution over running ZooKeeper/KRaft service instance:
   • kafka:zk://[ip]:[port] - for Apache Kafka VMs access. Stream-JMX will monitor /broker/ids node for registered brokers and will connect to all JMX enabled (exposing JMX port not equal -1) brokers.
   • solr:zk://[ip]:[port] - for Apache Solr VMs access. Stream-JMX will monitor /live_nodes node for registered Solr node instances. NOTE: Solr does not expose JMX port, so you need to specify JMX port number as Other options column value port=[port number].
   • kafka:kraft://[ip]:[port] or kraft://[ip]:[port] - for KRaft coordinated Apache Kafka VMs access.

NOTE: ZooKeeper service instance JMX sampling does not start automatically defining VM descriptors for Apache Kafka or Apache Solr. In case you want to sample JMX of referred ZooKeeper instance, you have to define ZooKeeper runner VM dedicated JMX service URL.

NOTE: When Stream-JMX binds to ZooKeeper node, it monitors for changes under that node. So if new brokers gets registered/unregistered, Stream-JMX connects/disconnects JMX services of these brokers on runtime.

NOTE: When Stream-JMX binds to KRaft controller broker, it checks for nodes changes periodically. So if new brokers gets registered/unregistered, Stream-JMX connects/disconnects JMX services of these brokers on runtime. Period is configured over connection custom scope property brokers.refresh.period.sec like this: kafka.custom.brokers.refresh.period.sec: 20 (every 20 seconds).

Additions needed to run SamplingAgent connected to remote WAS machine can be found in executable OS shell run script files bin/stream-jmx-connect-was.bat or bin/stream-jmx-connect-was.sh. It contains these major configuration additions:

• WAS environment setup configuration
• appending LIBPATH variable with WAS libs
• adding WAS specific JMX sampler options
• defining WAS JMXConnector parameters
• and adding them as additional java command parameters

When you have IBM JVM running on client side and getting authentication or naming related exceptions, you have to:

• Alter sas.client.props by setting com.ibm.CORBA.validateBasicAuth=false

• Add additional SamplingAgent arguments:

      -ul:[your WAS user]
      -up:[your WAS user pass]
      -cp:java.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
      -cp:java.naming.factory.url.pkgs=com.ibm.ws.naming
      -cp:java.naming.provider.url=corbaloc:iiop:localhost:2809/WsnAdminNameService

Additions needed to run SamplingAgent connected to remote WebLogic machine can be found in executable OS shell run script files bin/stream-jmx-connect.bat or bin/stream-jmx-connect.sh. It contains these major configuration additions:

• WebLogic environment setup configuration in setDomainEnv.sh/setDomainEnv.cmd file, exposing JMX access port and security options like:

  • .bat/cmd

    set "JAVA_OPTIONS=%JAVA_OPTIONS% -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8788 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

  • .sh

    JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8788 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
    export JAVA_OPTIONS

• Appending LIBPATH variable with WebLogic client libs from ${WL_INSTALL_DIR}/wlserver/server/lib/:

  • wlclient.jar
  • wljmxclient.jar
  • javax.javaee-api.jar
  • jakarta.jakartaee-api.jar

  For example:

  • .bat/cmd

    set "WL_HOME=C:\Oracle\Middleware\Oracle_Home"
    set "WL_CLIENT_LIBS=%WL_HOME%\wlserver\server\lib\wlclient.jar;%WL_HOME%\wlserver\server\lib\wljmxclient.jar;%WL_HOME%\wlserver\server\lib\javax.javaee-api.jar;%WL_HOME%\wlserver\server\lib\jakarta.jakartaee-api.jar"
    set "LIBPATH=%LIBPATH%;%RUNDIR%..\lib\*;%WL_CLIENT_LIBS%"

  • .sh

    WL_HOME="/opt/Oracle/Middleware/Oracle_Home"
    WL_CLIENT_LIBS="$WL_HOME/wlserver/server/lib/wlclient.jar:$WL_HOME/wlserver/server/lib/wljmxclient.jar:$WL_HOME/wlserver/server/lib/javax.javaee-api.jar:$WL_HOME/wlserver/server/lib/jakarta.jakartaee-api.jar"
    LIBPATH="$LIBPATH:$SCRIPTPATH/../lib/*:$WL_CLIENT_LIBS"

• Enable j2ee module use:

  • .bat/cmd

    set "MODULE_SET=core j2ee"

  • .sh

    MODULE_SET=("core" "j2ee")

• Add additional SamplingAgent arguments:

      -cp:jmx.remote.protocol.provider.pkgs=weblogic.management.remote
      # Add user credentials if needed
      -cp:java.naming.security.principal=user
      -cp:java.naming.security.credentials=password
      # Optional naming parameters
      #-cp:java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory
      #-cp:java.naming.provider.url=t3://localhost:7001

WebLogic 12c provides these JNDI Names for WebLogic MBean Servers:

• weblogic.management.mbeanservers.domainruntime - Domain Runtime MBean Server
• weblogic.management.mbeanservers.runtime - Runtime MBean Server
• weblogic.management.mbeanservers.edit - Edit MBean Server

You can find your WebLogic server exposed JMX Connectors in console output:

<May 10, 2019, 10:57:11,147 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.runtime.>
<May 10, 2019, 10:57:11,823 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.domainruntime.>
<May 10, 2019, 10:57:11,853 AM EEST> <Warning> <JMX> <BEA-149512> <JMX Connector Server started at service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.edit.>

JMX Service connection URL for WebLogic is like this:

service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.runtime

And complete set of stream-jmx program arguments:

-connect
-vm:service:jmx:iiop://172.16.6.56:7001/jndi/weblogic.management.mbeanservers.edit
-ao:*:*!!10000
-cp:jmx.remote.protocol.provider.pkgs=weblogic.management.remote
-cp:java.naming.security.principal=weblogic
-cp:java.naming.security.credentials=password

For more information, see Programming WebLogic JNDI, WebLogic API Environment class documentation and Accessing WebLogic Server MBeans with JMX.

You can connect SamplingAgent to JVM from your custom API by calling SamplingAgent.connect(String,String) method.

Sample connect call for local JVM:

try {
   SamplingAgent.newSamplingAgent().connect("activemq", "*:*!!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

Sample connect call for remote JVM:

try {
   SamplingAgent.newSamplingAgent().connect("service:jmx:iiop://172.16.6.40:2809/jndi/JMXConnector", "*:*!!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

Sample connect call for remote JVM defining username and password:

try {
   SamplingAgent.newSamplingAgent().connect("activemq", "admin", "admin", "*:*!!10000");
} catch (Exception exc) {
  exc.printStackTrace();
}

try {
  Map<String, Object> connParams = new HashMap<String, Object>();
  connParams.put("javax.net.ssl.trustStore", "/your/path/to/truststore.jks");
  connParams.put("javax.net.ssl.trustStorePassword", "truststore_pwd");
   SamplingAgent.newSamplingAgent().connect("activemq", "*:*!!10000", connParams);
} catch (Exception exc) {
  exc.printStackTrace();
}

java -Dtnt4j.config=./config/tnt4j.properties -Dlog4j2.configurationFile=./config/log4j2.xml -classpath "tnt4j-stream-jmx-core.jar;lib/*" com.jkoolcloud.tnt4j.stream.jmx.SamplingAgent -local -ao:*:*!*:dummy!10000

System properties -Dxxxxx define Stream-JMX configuration. For details see Stream-JMX configuration .

SamplingAgent arguments -connect -vm:activemq -ao:*:*!*:dummy!10000 states:

• -local - defines that SamplingAgent shall run sampling on local process runner JVM.
• -ao:*:*!*:dummy!10000 - is JMX sampler options stating to include all MBeans, exclude all dummy MBeans and schedule sampling every 10 seconds. Sampler options are optional - default value is *:*!30000. Initial sampler delay can be configured by adding numeric parameter *:*!30000!1000 defining initial sampler delay as 1 second. Default sampler delay value is equal to sampling period value. See JMX Sampling Agent sampler options for details.
• -slp: - any JMX sampler configuration property. See Program arguments used for details.
• -sp: - any JVM system property used by sampler. See System properties used for details. NOTE: This argument allows to use su.sh or su.cmd encoded property values (e.g. passwords).

You can run SamplingAgent for local process runner JVM from your custom API by calling SamplingAgent.sampleLocalVM(String,boolean) method.

Sample sampleLocalVM call for a local process runner JVM:

try {
  SamplingAgent.sampleLocalVM("*:*!!10000", true);
} catch (Exception exc) {
  exc.printStackTrace();
}