Configuring Stream-JMX Sampler Properties Using System and Program Arguments

Stream-JMX has configuration properties allowing to configure JMX sampler. It is possible to configure the same stream-JMX sampler parameter using System property or program argument. Depending on sampling environment used, in some cases, it is easier to configure it using one approach or another. When both definitions are available, the System property value is assigned first, and then the program argument value.

JMX sampler configuration properties are:

• forceObjectName - flag indicating to forcibly add objectName attribute if such is not present for a MBean. Default value: false.
• addStatisticMetadata - flag indicating to add J2EE metadata entries as attributes for a MBean. Default value: false.
• compositeDelimiter - delimiter used to tokenize composite/tabular-type MBean property keys. Default value: \;
• useObjectNameProperties - flag indicating to copy MBean ObjectName contained properties into sample snapshot properties. Default value: true.
• excludeOnError - flag indicating to auto-exclude failed-to-sample attributes. Default value: false.
• excludedAttributes - list of user-chosen attribute names (may have wildcards * and ?) to exclude, pattern: attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName. Default value: ``.

See System properties used to configure Stream-JMX using system properties. See Program arguments used how to configure Stream-JMX using program arguments.

To define system property for application you can use common JVM argument -Dkey=value or SamplingAgent program argument -sp:key=value.

General use:

• tnt4j.config - defines TNT4J properties file path. Example: -Dtnt4j.config="./config/tnt4j.properties"
• log4j2.configurationFile - defines stream-jmx logging used LOG4J properties file path. Example: -Dlog4j2.configurationFile="./config/log4j2.xml"
• tnt4j.stream.jmx.agent.forceObjectName - defines whether to forcibly add objectName attribute if such is not present for a MBean. Default value: false. Example: -Dtnt4j.stream.jmx.agent.forceObjectName=true
• tnt4j.stream.jmx.agent.addStatisticMetadata - defines whether to add J2EE statistic metadata entries as attributes for a MBean. Default value: false. Example: -Dtnt4j.stream.jmx.agent.addStatisticMetadata=true
• tnt4j.stream.jmx.agent.compositeDelimiter - defines delimiter used to tokenize composite/tabular-type MBean properties keys. Default value: \. Example: -Dtnt4j.stream.jmx.agent.compositeDelimiter=.
• tnt4j.stream.jmx.agent.useObjectNameProperties - defines whether to copy MBean ObjectName contained properties into sample snapshot properties. Default value: true. Example: -Dtnt4j.stream.jmx.agent.useObjectNameProperties=false
• sjmx.serviceId - defines stream-jmx service identifier used by TNT4J SourceFQN to distinguish monitored application instance. Example: -Dsjmx.serviceId=broker-0 or -sp:sjmx.serviceId=broker-0
• tnt4j.stream.jmx.agent.excludeOnError - defines whether to auto-exclude failed-to-sample attributes. Default value: false. Example: -Dtnt4j.stream.jmx.agent.excludeOnError=true
• tnt4j.stream.jmx.agent.excludedAttributes - defines list of user-chosen attribute names (may have wildcards * and ?) to exclude, pattern: attr1,attr2,...,attrN@MBean1_ObjectName;...;attr1,attr2,...,attrN@MBeanN_ObjectName. Default value: ``. Example: -Dtnt4j.stream.jmx.agent.excludedAttributes=javaVersion,javaVendor@WebSphere:mbeanIdentifier=cells*,*
• tnt4j.stream.jmx.sampler.factory- defines class name of SamplerFactory class to be used by stream. Default value: com.jkoolcloud.tnt4j.stream.jmx.factory.DefaultSamplerFactory. Example: -Dtnt4j.stream.jmx.sampler.factory=com.jkoolcloud.tnt4j.stream.jmx.impl.WASSamplerFactory
• tnt4j.stream.jmx.sampler.batch.period.sec - defines current batch completion period in seconds. Default value: 30 sec.
• tnt4j.stream.jmx.sampler.batch.used.memory.percent - defines current batch completion by used memory percentage threshold. Default value: 80%
• tnt4j.stream.jmx.sampler.mbeans.sync.rate - defines JMX MBean attributes sampling iterations count to run periodic synchronization of monitored MBeans. While JMX server runtime, dynamic MBeans may not emit client-side expected register/unregister notifications, and that way de-synchronization of monitored MBeans may appear. 0 means never. Default value: 0.
• tnt4j.stream.jmx.sampler.server.ready.uptime.sec - defines monitored JVM instance uptime in seconds required to consider it is ready for JMX MBean attributes sampling. Default value: 10 sec.
• tnt4j.stream.jmx.sampler.mbeans.query.retry.interval.sec - defines the retry interval (in seconds) for failed MBean name queries. Default value: 5 sec.
• tnt4j.stream.jmx.sampler.mbeans.query.retry.max.attempts.count - defines the maximum retry attempts count for failed MBean name queries. Default value: 3.
• tnt4j.stream.jmx.sampler.mbeans.register.delay - defines the delay in milliseconds to start processing MBean registration notification received from JMX server. There are some cases (like Kafka broker shutdown) when MBeans get into almost instantaneous register/unregister loop. To rule-out such "fake/temporal" registrations this property is used to define delay and ignore MBean registrations followed by almost immediate unregister event. Default value: 1500.
• tnt4j.stream.jmx.sampler.task.thread.count - defines thread count used by sample executor to perform MBean attributes sampling. Default value: 5.
• tnt4j.stream.jmx.sampler.task.offer.timeout.sec - defines timeout in seconds to offer MBean attributes sample task to be accepted by sample executor. Default value: 60 sec.
• tnt4j.stream.jmx.sampler.poll.timeout.msec - defines timeout in milliseconds for MBean attributes sample poll from sample executor queue. Default value: 500 ms.
• tnt4j.stream.jmx.sampler.thread.count - defines thread count used by sampler to run MBeans sampling iteration. It must be multiple of 2: one thread for MBean attributes collection running parallel sampling tasks, the other for obtained MBean attributes processing. Default value: 2.
• tnt4j.stream.jmx.formatter.fact.group.type.name - defines facts group type name formatter adds to facts batch (empty value makes skipping to set this value). Default value: Activities for ACTIVITY type batches, and Events for EVENT type batches.
• tnt4j.stream.log.filename - defines name of stream log file. Default value: ./logs/tnt4j-stream-jmx.log. Example: -Dtnt4j.stream.log.filename=./logs/tnt4j-stream-jmx_broker0.log
• tnt4j.activities.log.filename - defines name of streamed activities log file. Default value: ./logs/tnt4j-stream-jmx_samples.log. Example: -Dtnt4j.activities.log.filename=./logs/tnt4j-stream-jmx_broker0_samples.log
• tnt4j.stream.jmx.include.filter - MBean include name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter (domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3) or grouping using () and | notation. Grouping like java.lang:type=(Threading|Memory|OperatingSystem|Runtime) is equal to java.lang:type=Threading;java.lang:type=Memory;java.lang:type=OperatingSystem;java.lang:type=Runtime;. Default value: *:*.
• tnt4j.stream.jmx.exclude.filter - MBean exclude name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter (domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3) or grouping using () and | notation. Grouping like java.lang:type=(Threading|Memory|OperatingSystem|Runtime) is equal to java.lang:type=Threading;java.lang:type=Memory;java.lang:type=OperatingSystem;java.lang:type=Runtime;. Default value: ``.
• tnt4j.stream.jmx.period - MBeans sampling rate in milliseconds. Default value: 30 sec. 
  NOTE: if MBean sampling duration gets longer than configured, it will dynamically reschedule sampling from fixed rate to fixed delay in sample-ms between samples.
• tnt4j.stream.jmx.init.delay - MBeans sampling initial delay in milliseconds. Default value: equal to tnt4j.stream.jmx.period value.
• tnt4j.stream.jmx.batch.size - number of sampled MBeans to post over single package. -1 means unlimited batch - all MBeans from sampling iteration are packed into single package. Default value: 20. NOTE: if current MBean sampling iteration takes longer than system property tnt4j.stream.jmx.sampler.batch.period.sec defined value or used memory consumption percentage gets higher than system property tnt4j.stream.jmx.sampler.batch.used.memory.percent defined value, current batch is also considered to be complete.
• tnt4j.stream.jmx.time.units - defines name of time units to be used for stream used periods and delays. Default value: MILLISECONDS.
• tnt4j.stream.jmx.post.snapshots.only - defines flag indicating to send only SNAPSHOT entities without wrapping ACTIVITY type entity. Default value: false.
• tnt4j.stream.jmx.scheduler.lagging.period.sec - defines period in seconds for sampling scheduler to switch to, when sampling iteration execution time gets longer than period between two consecutive sample iterations triggering (sampler starts lagging). Scheduler then switches from FIXED_RATE to FIXED_DELAY mode. When sample iterations execution starts to fit into main scheduler period again - scheduler switches back to main period and FIXED_RATE mode. Default value: 10 sec..

Changes between versions:

• Prior to version 0.7 stream-jmx was writing logging messages to System.out/err print streams. Since version 0.7 logging is performed over TNT4J Log Sink to slf4j-log4j12. Logger configuration is defined in ./config/log4j2.xml file. If you were using system property -Dtnt4j.stream.jmx.agent.trace=true prior to 0.7 version, this is now configured over logger log level, by setting it to DEBUG value, e.g:

  <Logger name="com.jkoolcloud.tnt4j.stream.jmx" level="DEBUG"/>

When defining configuration file properties over file: URI notation (e.g. configurationFile for log4j or logback) on Windows change backslashes \  to slash / symbols for URI to get successfully parsed.

When running multiple parallel instances of stream-jmx (e.g. collecting Kafka broker metrics using dedicated stream instance), it is recommended to define different log file names (using system properties tnt4j.stream.log.filename and tnt4j.activities.log.filename) for every stream-jmx running JVM. Using same log file on different VMs may cause logging conflicts when a VM logger holds different last log line references, resulting log entries to be written in random order. When monitoring multiple VM's over single stream-jmx instance, only one logger instance is run (as there is one JVM running), and no logging conflicts shall occur.

To define stream JMX sampler configuration property use program argument-slp:. One argument defines one property. To define multiple properties use as many argument definitions as there are required properties. For example:

-slp:forceObjectName=true
-slp:addStatisticMetadata=true
-slp:compositeDelimiter=.
-slp:useObjectNameProperties=false
-slp:excludeOnError=true
-slp:excludedAttributes=java*@WebSphere:mbeanIdentifier=cells*,*

Agent options are defined using format: mbean-filter!exclude-filter!sample-ms!init-delay-ms!batch-size

• mbean-filter - MBean include name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter (domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3) or grouping using () and | notation. Grouping like java.lang:type=(Threading|Memory|OperatingSystem|Runtime) is equal to java.lang:type=Threading;java.lang:type=Memory;java.lang:type=OperatingSystem;java.lang:type=Runtime;.
• exclude-filter - MBean exclude name filter defined using object name pattern: domainName:keysSet. NOTE: Multiple filters can be defined using ; as delimiter (domainName1:keysSet1;domainName2:keysSet2;domainName3:keysSet3) or grouping using () and | notation. Grouping like java.lang:type=(Threading|Memory|OperatingSystem|Runtime) is equal to java.lang:type=Threading;java.lang:type=Memory;java.lang:type=OperatingSystem;java.lang:type=Runtime;.
• sample-ms - MBeans sampling rate in milliseconds. NOTE: if MBean sampling duration gets longer than configured, it will dynamically reschedule sampling from fixed rate to fixed delay in sample-ms between samples.
• init-delay-ms - MBeans sampling initial delay in milliseconds. Optional, by default it is equal to sample-ms value.
• batch-size - number of sampled MBeans to post over single package. -1 means unlimited batch - all MBeans from sampling iteration are packed into single package. Optional, default is 20. NOTE: if current MBean sampling iteration takes longer than system property tnt4j.stream.jmx.sampler.batch.period.sec defined value or used memory consumption percentage gets higher than system property tnt4j.stream.jmx.sampler.batch.used.memory.percent defined value, current batch is also considered to be complete.

Default sampling agent options value is: *:*!!30000!30000!20

Stream-JMS has a couple of additional features in comparison with basic TNT4J when building Source RootFQN value:

1. Has two dedicated value placeholders sjmx.serverAddress and sjmx.serverName to resolve remotely sampled machine IP address and host name. To map these sampled machine IP/host name values into source field, put @ symbol before placeholder name in source field definition. In this case tnt4j.properties configuration would be like this:

   ; Remote machine IP address
   ...
   source.factory.GENERIC: Streams
   source.factory.DATACENTER: HQDC
   source.factory.SERVICE: broker-01
   source.factory.SERVER: @sjmx.serverAddress
   source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?
   ...
   
   ; Remote machine host name
   ...
   source.factory.GENERIC: Streams
   source.factory.DATACENTER: HQDC
   source.factory.SERVICE: broker-01
   source.factory.SERVER: @sjmx.serverName
   source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?
   ...

    

2. Can resolve values for source fields from JMX MBean attributes. Two items are required to use in tnt4j.properties configuration to enable this feature:

   1. Set source.factory to com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
   2. Put MBean attribute descriptor having prefix @bean: as source field value. There are two types of MBean attribute descriptors:
      • Mbean attribute value descriptor pattern is "@bean:MBean_ObjectName/?AttributeName. It also allows use of wildcard symbols *, e.g.: @bean:org.apache.ZooKeeperService:name0=*/?ClientPort resolving ClientPort value for first available (if more than one is running on same JVM, but usually there should be only one) ZooKeeper service instance.
      • MBean key property descriptor pattern is @bean:MBean_ObjectName:PropertyKey=?(,OtherProperties) (part within () is optional), e.g.: @bean:kafka.server:id=?,type=app-info.
   3. Since MBean initialization may take some time on server side (usually it may happen after some server component restart) MBean attribute value may not be resolvable at that time. To overcome such a case, you can define MBean value resolution options:
      • RetryIntervalSec - retry interval in seconds. Default value: 1 sec..
      • RetryMaxAttempts - maximum retry attempts count. Default value: 5.
      • ExpIntervalAfter - retry attempt count at which exponential interval growth begins. 0 means keep using base interval defined by RetryMaxIntervalSec. Default value: 0.
      • RetryMaxIntervalSec - maximum retry interval in seconds. Works in pair with ExpIntervalAfter. Default - interval defined by RetryIntervalSec option.

   then, putting all together, thetnt4j.properties configuration will like this:

   ...
   source.factory: com.jkoolcloud.tnt4j.stream.jmx.source.JMXSourceFactoryImpl
   source.factory.GENERIC: Streams
   source.factory.DATACENTER: HQDC
   source.factory.SERVICE: @bean:org.apache.activemq:type=Broker,brokerName=localhost/?BrokerId
   source.factory.SERVER: @bean:JMImplementation:type=MBeanServerDelegate/?MBeanServerId
   source.factory.RootFQN: SERVICE=?#SERVER=?#DATACENTER=?#GENERIC=?
   source.factory.RetryIntervalSec: 1
   source.factory.RetryMaxAttempts: 10
   source.factory.ExpIntervalAfter: 0
   source.factory.RetryMaxIntervalSec: 120
   ...