TNT4J-Streams

This article shows how to use TNT4J-Streams to stream data from different sources, configure it, and run it in various ways.

tnt4j-streams is under the Nastel Technologies EULA.

• TNT4J-Streams can be run out of the box for a large set of data streaming without writing any additional code. All you need is to define your data format mapping to TNT4J event mapping in TNT4J-Streams configuration.

• Supports the following data sources:

  • File
  • Characters/bytes feed stream from a file or over TCP/IP
  • HDFS
  • MQTT
  • HTTP
  • JMS
  • Apache Kafka (as Consumer and as Producer/Consumer interceptor)
  • Apache Flume
  • Logstash
  • WMQ (IBM MQ)
  • OS pipes
  • Zipped files (also applies to HDFS)
  • Standard Java InputStream/Reader
  • JAX-RS service (JSON/XML)
  • JAX-WS service
  • System command
  • MS Excel document
  • Elastic Beats
  • FileSystem (JSR-203 compliant) provided files (accessing remote files over SCP/SSH, SFTP, etc.)
  • JDBC
  • Chronicle Queue
  • Artemis Broker or Producer/Consumer interceptor
  • Compressed binary data or input stream
  • Protobuf messages

• Files (including provided by HDFS and JSR-203 FileSystem) can be streamed:

  • as "whole at once" - when a stream starts, it reads file contents line by line, meaning each file line holds the data for a single activity event. After file reading completes, the stream stops.
  • using file polling - when an application writes data to a file at runtime, the stream waits for file changes. When the file changes, appended lines are read by stream and interpreted so that each line represents a single activity event. The stream stops only when the application is terminated or some critical runtime error occurs.

• Customized parser for Apache Access Logs.

• Customized IBM MQ Trace Events stream (and parser).

• Customized IBM MQ Error Log parser (also supports JSON-formatted logs).

• It can be integrated with:

  • Logstash
  • Apache Flume
  • Angulartics
  • AJAX
  • Node.js
  • Collectd
  • Nagios
  • Elastic Beats
  • AWS CloudWatch metrics over Kinesis FireHose
  • Prometheus Remote-Write over Snappy compressed protobuf messages

  just by applying configuration, without additional coding.

• Redirect streamed data from different TNT4J-based producer APIs like tnt4j-stream-* to a TNT4J-based streams concentrator.

• Run TNT4J-Streams as a system daemon service.

• Run TNT4J-Streams as an HTTP POST request consumer servlet.

• Supports XPath 1.0, XPath 2.0, XPath 3.0 and XPath 3.1 expressions for XML parsing.

• Select File->Import...->Maven->Existing Maven Projects
• Click 'Next'
• In the 'Root directory' field, select the path of the directory where you downloaded the TNT4J-Streams project.
• Click 'OK'
• Dialog fills in with project modules details
• Click 'Finish'

• As a standalone application:

  • Write a streams configuration file. See Streams configuration for more details
  • Configure your loggers
  • Use bin/tnt4j-streams.bat or bin/tnt4j-streams.sh to run the standalone application

  NOTE: If you are using Java 9 as your runtime JVM and encounter java.lang.NoClassDefFoundError: javax/xml/bind/JAXBException, add the java command parameter --add-modules java.xml.bind to include JAXB classes in the classpath.

• As an API integrated into your product:

  • Use the following Maven dependency:

        <dependency>
            <groupId>com.jkoolcloud.tnt4j.streams</groupId>
            <artifactId>tnt4j-streams-core</artifactId>
            <version>12.0.0</version>
        </dependency>

  • Write a streams configuration file. See Streams configuration for more details.
  • Use StreamsAgent.runFromAPI(new CfgStreamsBuilder().setConfig(configFileName)) in your code

• As a system daemon service. See TNT4J-Streams as System Service configuration for details on how to run TNT4J-Streams as a system service.

• Utility operations:

  • To encrypt a plain-text password, use bin/utils/su.cmd or bin/utils/su.sh. For example:

    *nix

    ./bin/utils/su.sh -e somePass

    Windows

    \bin\utils\su.cmd -e somePass

    The output of this command looks like this: ENCRYPTED PASSWORD: v5OmKh7o3bHWRibcb4bnLg==. 
    You can now paste the encrypted password value (e.g., v5OmKh7o3bHWRibcb4bnLg) into your stream configuration.