TNT4J - Track and Trace API for Java – meshIQ

TNT4J is about tracking and tracing applications, activities, transactions, behavior, and performance via an easy-to-use API that behaves much like a logging framework.

Why Track and Trace Your Applications?

Track application behavior and performance to improve diagnostics
Track end-user behavior to improve usability and customer satisfaction
Track topology, communications, and relationships between entities
Track messages — binary, text, JSON, XML, video, image, voice
Track location, mobility, and GPS of your applications and users
Track anything worth tracking in your application
Know your application: what, where, when, why

Events may have a TTL (time-to-live) for sinks that support event expiration.
An event rate limiter (throttle control) is available based on MPS or BPS (messages/sec, bytes/sec).

Several key features make TNT4J a prime choice for your java application:

Quick Examples

Use Maven dependency:

    <dependency>
        <groupId>com.jkoolcloud</groupId>
        <artifactId>tnt4j-core</artifactId>
        <version>12.0.2</version>
    </dependency>

Basic Logging Example

TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
try {
   ...
} catch (Exception e) {
   logger.error("Failed to process request={}", request_id, ex);
}

Using Correlator and Relative Class Marker

Below is an example using a correlator and a relative class marker to locate the caller's method call on the stack — $my.package.myclass:0.
This marker will resolve to the caller’s method name above all the calls that start with $my.package.myclass.

TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
try {
   ...
} catch (Exception e) {
   logger.tnt(OpLevel.ERROR, "$my.package.myclass:0", myCorrelator, 
       "Failed to process request={}", request_id, ex);
}

Conditional Logging with an Event Filter

You can consolidate all conditional logging checks into a single listener.
Why call isDebugEnabled() before each log entry?

TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
logger.addSinkEventFilter(new MyEventFilter(logger));
try {
   logger.debug("My debug message {}, {}", arg0, arg1); // no need to gate this call
   ...
} catch (Exception e) {
   logger.error("Failed to process request={}", request_id, ex);
}

class MyEventFilter implements SinkEventFilter {
    TaskLogger logger;

    MyEventFilter(TaskLogger lg) {
        logger = lg;
    }

    @Override
    public boolean filter(EventSink sink, TrackingEvent event) {
        return logger.isSet(event.getSeverity(), "myappl.token");
    }

    @Override
    public boolean filter(EventSink sink, TrackingActivity activity) {
        return logger.isSet(activity.getSeverity(), "myappl.token");
    }

    @Override
    public boolean filter(EventSink sink, Snapshot snapshot) {
        return logger.isSet(snapshot.getSeverity(), "myappl.token");
    }

    @Override
    public boolean filter(EventSink sink, long ttl, Source src, OpLevel level, String msg, Object... args) {
        return logger.isSet(level, "myappl.token");
    }
}

Get Started Quickly

Embed TNT4J into your application and realize the benefits in a matter of minutes.
TNT4J can also integrate with other lower-level logging frameworks such as slf4j, log4j and JUL
The default TNT4J binding is based on JUL.

TNT4J Stream Configuration

TNT4J stream configuration is defined in the tnt4j.properties file, which is located by specifying the tnt4j.config Java property (e.g., -Dtnt4j.config=./config/tnt4j.properties).

If the TNT4J configuration is split into separate external files, use the tnt4j.config.path Java property to define the root path common to all these files (e.g., -Dtnt4j.config.path=./config/). Then, tnt4j.properties file-defined import entries will be treated as paths relative to the tnt4j.config.path value (e.g., import: tnt4j-common.properties becomes ./config/tnt4j-common.properties).

By default, the common configuration file root path is resolved in the following order:

Path defined using the system property tnt4j.config.path
Parent directory of the file defined by the tnt4j.config system property
Default path ./config

This configuration file defines all event sources and target event sink bindings (e.g., File, MQTT, Kafka, HTTPS), as well as stream-specific attributes. Here's an example of a sample stream configuration:

; TNT4J Common Definitions
{
    ; import common tnt4j logger definitions
    source: common.base
    import: tnt4j-common.properties
}
;Stanza used for sources that start with com.jkoolcloud
{
    source: com.myappl
    like: default.logger
    source.factory.RootSSN: tnt4j-samples

    tracker.default.snapshot.category: DefaultCategory
    event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
    event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory
    event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl
    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: INFO
    event.formatter: com.jkoolcloud.tnt4j.format.SimpleFormatter
    activity.listener: com.jkoolcloud.tnt4j.tracker.DefaultActivityListener
}
;Stanza used for sources that start with org
{
    source: org
    ; import settings from com.myappl stanza
    like: com.myappl
}

The import property allows defining an absolute file path or a relative file path from the common configuration files root path. If some files are located outside this root path, you can use the import.path property to define a specific path for the imported file within the same stanza:

; TNT4J Common Definitions
{
    ; import common tnt4j logger definitions
    source: common.base
    import: tnt4j-common.properties
    import.path: ../../personal-config/
}

Stream Over Socket

You can stream your events over a socket using the com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory event sink factory.

To specify the socket type, use host prefixes:

tcp: – Plain TCP socket. For example: tcp:localhost.
Note: Omitting the socket type prefix defaults to tcp:, so localhost is treated as tcp:localhost.
tcps: – SSL-wrapped TCP socket. For example: tcps:localhost.
udp: – Plain UDP (datagram) socket. For example: udp:localhost.

Configure the event sink in tnt4j.properties as follows:

...
; Use buffered event sink around Socket event sink factory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
# "Endpoint" property has alias named "Address"
event.sink.factory.EventSinkFactory.Endpoint: localhost:6408
# Socket connection timeout in seconds 
event.sink.factory.EventSinkFactory.ConnTimeout: 30
# Socket read timeout in seconds
event.sink.factory.EventSinkFactory.ReadTimeout: 10
# Flag indicating whether to add UDP message index "header" (first 6 bytes of transmitted data having pattern &XXXX$, where XXXX is sequence number)
# when transmitting large payload in chunks
event.sink.factory.EventSinkFactory.AddUDPMsgIndex: false
...

SocketEventSinkFactory supports multiple fallback endpoints, separated by commas. Only one is active at a time; others are used if the current one becomes unavailable.

event.sink.factory.EventSinkFactory.Endpoint: localhost:6005,tcp:host1:6005,udp:localhost:6006

Using Proxy (SOCKSv5)

Socket streaming can also be combined with proxy use. Proxy authentication properties (ProxyUser, ProxyPass) are optional. To configure:

...
; Use buffered event sink around Socket event sink factory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
event.sink.factory.EventSinkFactory.Endpoint: localhost:6408
event.sink.factory.EventSinkFactory.ProxyHost: proxy.host.com
event.sink.factory.EventSinkFactory.ProxyPort: 6666
# Optional proxy properties
event.sink.factory.EventSinkFactory.ProxyUser: proxy-user
event.sink.factory.EventSinkFactory.ProxyPass: proxy-pass
...

Proxy authentication credentials can also be passed through JVM system properties:

-Djava.net.socks.username=proxy-user -Djava.net.socks.password=proxy-pass

Refer to Java SE documentation for more details on using system properties for proxy configuration.

Stream Over Kafka

Use Maven dependency:

    <dependency>
        <groupId>com.jkoolcloud</groupId>
        <artifactId>tnt4j-kafka-sink</artifactId>
        <version>12.0.2</version>
    </dependency>

Use the KafkaEventSinkFactory to stream events over Apache Kafka.

Configure event sink in tnt4j.properties as follows:

...
; Use buffered event sink around Kafka event sink factory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.kafka.KafkaEventSinkFactory
event.sink.factory.EventSinkFactory.propFile: <your path>/tnt4j-kafka.properties
event.sink.factory.EventSinkFactory.topic: tnt4j-topic
event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter
...

Sample tnt4j-kafka.properties file:

bootstrap.servers=localhost:9092
acks=all
retries=0
linger.ms=1
buffer.memory=33554432
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer

Stream Over MQTT

Use Maven dependency:

    <dependency>
        <groupId>com.jkoolcloud</groupId>
        <artifactId>tnt4j-mqtt-sink</artifactId>
        <version>12.0.2</version>
    </dependency>

Configure MQTT streaming using MqttEventSinkFactory:

...
; Use buffered event sink around MQTT event sink factory
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl

event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.mqtt.MqttEventSinkFactory
event.sink.factory.mqtt-server-url: tcp://localhost:1883
event.sink.factory.mqtt-topic: tnt4jStream
event.sink.factory.mqtt-user: mqtt-user
event.sink.factory.mqtt-pwd: mqtt-pwd
...

SLF4J Event Sink Integration

TNT4J offers default logging integration with SLF4J via:

com.jkoolcloud.tnt4j.sink.impl.slf4j.SLF4JEventSinkFactory

Other logging frameworks can be supported by implementing the EventSinkFactory and EventSink interfaces.

All TNT4J messages can be routed via an SLF4J event sink and therefore can take advantage of the underlying logging frameworks supported by SLF4J.

Developers can also enrich event messages and pass context to TNT4J using the hashtag enrichment scheme. Hashtags decorate event messages with important metadata about each log message. This metadata is used to generate TNT4J tracking events:

logger.info("Starting a tnt4j activity #beg=Test, #app=" + Log4JTest.class.getName());
logger.warn("First log message #app=" + Log4JTest.class.getName() + ", #msg='1 Test warning message'");
logger.error("Second log message #app=" + Log4JTest.class.getName() + ", #msg='2 Test error message'", new Exception("test exception"));
logger.info("Ending a tnt4j activity #end=Test, #app=" + Log4JTest.class.getName() + " #%i/order-no=" + orderNo  + " #%d:currency/amount=" + amount);

The example above groups messages between the first and last into a related logical collection called an Activity. An Activity is a collection of logically related events/messages. The hashtags #beg and #end are used to demarcate activity boundaries. This method also supports nested activities.

User-defined fields can be reported using the #[data-type][:value-type]/your-metric-name=your-value convention (e.g., #%i/order-no=62627 or #%d:currency/amount=50.45).

TNT4JAppender supports the following optional data-type qualifiers:

    %i/ -- integer
    %l/ -- long
    %d/ -- double
    %f/ -- float
    %b/ -- boolean
    %n/ -- number
    %s/ -- string

All value-type qualifiers are defined in com.jkoolcloud.tnt4j.core.ValueTypes. For examples:

currency     -- generic currency
    flag         -- boolean flag
    age         -- age in time units
    guid         -- globally unique identifier
    guage        -- numeric gauge
    counter        -- numeric counter
    percent        -- percent
    timestamp    -- timestamp
    addr         -- generic address

If no qualifier is specified, TNT4JAppender performs auto-detection: it first tests whether the value is numeric; if that test fails, it defaults to a string (e.g., #order-no=62627).

JUL (java.util.logging) Event Sink Integration

Use the JVM system property java.util.logging.config.file to define the JUL configuration file (e.g., logging.properties):

    -Djava.util.logging.config.file=./config/logging.properties

Performance

There is no need to concatenate messages before logging. String concatenation is expensive, especially in loops. Instead, log using message patterns, and TNT4J will resolve the message only if it actually gets logged:

logger.debug("My message {}, {}, {}", arg0, arg1, arg3);

TNT4J enhances logging performance by supporting asynchronous pooled logging, which delegates logging to a dedicated thread pool. Use BufferedEventSinkFactory in your tnt4.properties configuration to enable this feature:

...
event.sink.factory: com.jkoolcloud.tnt4j.sink.impl.BufferedEventSinkFactory
event.sink.factory.PooledLoggerFactory: com.jkoolcloud.tnt4j.sink.impl.PooledLoggerFactoryImpl
event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.slf4j.SLF4JEventSinkFactory
...

Simplicity & Clean Code

There is no need to check for isDebugEnabled() before logging messages. Simply register your own SinkEventFilter and consolidate all checking into a single listener:

logger.addSinkEventFilter(new MyLogFilter()); 
...
logger.debug("My message {}, {}, {}", arg0, arg1, arg2);

All conditional logging can be consolidated into a single listener object.

Flexible Filtering

Filter not only on category/severity (as in SLF4J) but also on performance objectives. For example, log events only when their elapsed or wait time exceeds a threshold. TNT4J lets you register filters in tnt4j.properties without changing application code. You can create custom filters to exclude events based on user-defined criteria and inject them via tnt4j.properties. See tnt4j.properties and com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter for details. Register filters via declarations in tnt4j.properties or programmatically in your application by creating your own event filter.

logger.addSinkEventFilter(new ThresholdEventFilter(OpLevel.WARNING));

Below is an example of an event-sink filter, ThresholdEventFilter, which must implement the SinkEventFilter interface.

public class ThresholdEventFilter implements SinkEventFilter {
    OpLevel threshold = OpLevel.INFO;

    public ThresholdEventFilter() {
    }

    public ThresholdEventFilter(OpLevel level) {
        this.threshold = level;
    }

    @Override
    public boolean filter(EventSink sink, TrackingEvent event) {
        return (event.getSeverity().ordinal() >= threshold.ordinal()) && sink.isSet(event.getSeverity());
    }

    @Override
    public boolean filter(EventSink sink, TrackingActivity activity) {
        return (activity.getSeverity().ordinal() >= threshold.ordinal()) && sink.isSet(activity.getSeverity());
    }

    @Override
    public boolean filter(EventSink sink, Snapshot snapshot) {
        return (snapshot.getSeverity().ordinal() >= threshold.ordinal()) && sink.isSet(snapshot.getSeverity());
    }

    @Override
    public boolean filter(EventSink sink, long ttl, Source source, OpLevel level, String msg, Object... args) {
        return (level.ordinal() >= threshold.ordinal()) && sink.isSet(level);
    }
}

Granular Conditional Logging

Log only what matters. Improve application performance by reducing excessive logging while increasing the relevance and quality of log output:

if (logger.isSet(OpLevel.DEBUG)) {
    logger.debug("My message {}, {}, {}", arg0, arg1, arg2);
}
if (logger.isSet(OpLevel.DEBUG, "myapp.mykey", myvalue)) {
    logger.debug("My message {}, {}, my.value={}", arg0, arg1, myvalue);
}

Checking a global debug flag is not granular enough for most applications. Many Java apps require fine-grained tracking to log only what matters based on specific context. Consolidate conditional checks (e.g., logger.isSet()) into a SinkEventFilter implementation and register it with the tracker instance.

logger.addSinkEventFilter(new MyLogFilter());

Share Logging Context Across Applications

Pass logging context across threads or applications:

logger.set(OpLevel.DEBUG, "myapp.mykey", myvalue);

Imagine you’re writing an application that must pass a tracking context/flag to downstream applications. TNT4J lets you set and get conditional variables within—and across—application boundaries.

Set and check tracking context as follows (track only requests matching a specific ZIP code):

// set level, key & value pair
logger.set(OpLevel.DEBUG, "zip-code", trackZipCode);
..
..
// check for sev, key & value pair match
String zipCode = request.getZipCode();  // example request containing a zip code
if (logger.isSet(OpLevel.DEBUG, "zip-code", zipCode)) {
    // your conditional logic here
}

State Logging

Log application state to improve the diagnosis of performance issues, resource utilization, and other hard-to-trace problems that standard event logging can miss. Simply register your state dump provider (see the DumpProvider interface) and export state variables specific to your application. Dump providers can be invoked on VM shutdown or on demand.

Generate an application dump on demand.

// register your dump provider
TrackingLogger.addDumpProvider(new MyDumpProvider());
...
TrackingLogger.dumpState();

Measurements & Metrics

TNT4J is not just about logging messages; it’s also about measurements and metrics—such as response time, CPU, memory, and block/wait times—as well as user-defined metrics. TNT4J lets you report metrics at the time of the logged event. Below is an example of creating a snapshot (a collection of metrics) and attaching it to an activity:

// post-processing of activity: enrich activity with application metrics
TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
TrackingActivity activity = logger.newActivity(OpLevel.INFO, "Order");
...
PropertySnapshot snapshot = logger.newSnapshot("Order", "Payment");
snapshot.add("order-no", orderNo, ValueTypes.VALUE_TYPE_ID);
snapshot.add("order-amount", orderAmount, ValueTypes.VALUE_TYPE_CURRENCY);
activity.add(snapshot); // add property snapshot associated with this activity
...
logger.tnt(activity); // report activity and associated snapshots as a single entity

A Snapshot is a collection of name–value pairs called a Property. Each Property can be further qualified with a value type defined in the ValueTypes class.

Below is an example of reporting a snapshot:

// post-processing of activity: enrich activity with application metrics
TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
TrackingActivity activity = logger.newActivity(OpLevel.INFO, "Order");
...
PropertySnapshot snapshot = logger.newSnapshot("Order", "Payment");
snapshot.add("order-no", orderNo, ValueTypes.VALUE_TYPE_ID);
snapshot.add("order-amount", orderAmount, ValueTypes.VALUE_TYPE_CURRENCY);
activity.tnt(snapshot); // add and report property snapshot associated with this activity

Below is an example of reporting standalone snapshot:

// post-processing of activity: enrich activity with application metrics
TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
...
PropertySnapshot snapshot = logger.newSnapshot("Order", "Payment");
snapshot.add("order-no", orderNo, ValueTypes.VALUE_TYPE_ID);
snapshot.add("order-amount", orderAmount, ValueTypes.VALUE_TYPE_CURRENCY);
logger.tnt(snapshot); // report a property snapshot

Correlation, Topology, Time Synchronization

Developers can relate events by grouping them into activities (an activity is a collection of related events and sub-activities) or by passing context correlators. Activity grouping and correlators create connectivity between events across thread, application, server, runtime, and location boundaries. TNT4J allows correlators to be attached when reporting tracking events—see TrackingLogger.tnt(..) for details. The same mechanism also supports relating tracking events across application and runtime boundaries.

TrackingLogger.tnt(..) also lets developers specify the flow of messages using the OpType.SEND and OpType.RECEIVE modifiers. These modifiers define message flow and direction, which is especially useful for applications that exchange information via networks, middleware, messaging, or other communication mechanisms. Tracking events annotated with these modifiers capture graph/topology information needed for root-cause analysis and for visualizing message flow.

Below is an example of a sender application:

// post-processing of activity: enrich activity with application metrics
TrackingLogger logger = TrackingLogger.getInstance(this.getClass());

// report sending an order with a specific correlator (order_id)
logger.tnt(OpLevel.INFO, OpType.SEND, "SendOrder", order_id, 
    elapsed_time, "Sending order to={}", destination);
// sending logic
....
....

Here is an example of a receiver application:

// post-processing of activity: enrich activity with application metrics
TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
...
// report received an order with a specific correlator (order_id)
logger.tnt(OpLevel.INFO, OpType.RECEIVE, "ReceiveOrder", order_id,
    elapsed_time, "Received order from={}", source);

TNT4J uses NTP natively to synchronize time across servers, enabling cross-server event correlation. To enable NTP time synchronization, define the Java system property: -Dtnt4j.time.server=ntp-server:123.

Use TimeServer.currentTimeMillis() instead of System.currentTimeMillis() to obtain NTP-adjusted time. TNT4J also maintains a microsecond-resolution clock via Useconds.CURRENT.get(), which returns the number of microseconds since midnight, January 1, 1970 (UTC, NTP-adjusted). TNT4J automatically measures and adjusts clock drift among NTP time, System.currentTimeMillis(), and System.nanoTime() to ensure accurate microsecond-level timing across VMs, devices, servers, and geolocations.

Tracking Associations

TNT4J allows developers to track associations between sources. A source is a logical definition of an entity such as an application, server, network, or geolocation. For example:
APP=WebAppl#SERVER=MYSERVER#DATACENTER=DC1#GEOADDR=New York
This means the application WebAppl is deployed on server MYSERVER, in data center DC1, located in New York.

Suppose you want to track an association between two applications that exchange data, where one application sends data to another:

// post-processing of activity: enrich activity with application metrics
TrackingLogger logger = TrackingLogger.getInstance(this.getClass());
...
TrackingEvent event = logger.newEvent(...);
// create an association between 2 applications
event.relate2(DefaultSourceFactory.getInstance().newFromFQN("APP=Orders#SERVER=HOST1#DATACENTER=DC1#GEOADDR=New York, NY"),
    DefaultSourceFactory.getInstance().newFromFQN("APP=Billing#SERVER=HOST2#DATACENTER=DC1#GEOADDR=London, UK",
    OpType.SEND);
logger.tnt(event);

The snippet above creates the association Orders->SEND->Billing which expresses the flow between the Orders and Billing applications, where Orders and Billing are nodes and SEND is the edge.

Logging Statistics

TNT4J maintains detailed statistics about logging activity. Each logger instance tracks counts of logged events, messages, errors, overhead (in microseconds), and more. This helps you understand the overhead of your logging framework on your application.

Obtain a map of all available key/value pairs:

Map<String, Long> stats = logger.getStats();
System.out.println("Logger stats: " + stats);
...
System.out.println("Resetting logger stats");
logger.resetStats();

Obtain metrics for all available trackers:

List<TrackingLogger> loggers = TrackingLogger.getAllTrackers();
for (TrackingLogger lg: loggers) {
    Map<String, Long> stats = lg.getStats();
    printStats(stats); // your call to print out tracker statistics
    ...
}

TNT4J also keeps stack traces for all TrackingLogger allocations. Here’s how to obtain stack frames for a set of TrackingLogger instances:

// obtain all available tracker instances
List<TrackingLogger> loggers = TrackingLogger.getAllTrackers();
for (TrackingLogger lg: loggers) {
    StackTraceElement[] stack = TrackingLogger.getTrackerStackTrace(lg);
    Utils.printStackTrace("Tracker stack trace", stack, System.out);
    ...
}