Skip to main content
Top
Review meshIQ Products (G2.com)

Sample Configuration: Stream and Parser Setup

This article shows a sample configuration for stream and parser setup, detailing how to configure the relevant components and parse incoming data entries.

System Command

Windows

This sample shows how to stream responses from executed Windows OS command.

Sample files can be found in samples/win-cmd-stream directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-ws.xsd">

    <parser name="ResponseParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityRegExParser">
        <property name="Pattern"
                  value="(\s*)&quot;\((.*)\)&quot;,&quot;(.*)&quot; &quot;(.*)&quot;,&quot;(.*)&quot;(.*)"/>

        <field name="EventType" value="SNAPSHOT"/>
        <field name="ProcessorTime" locator="5" locator-type="Index"/>
    </parser>

    <stream name="WinCmdStream" class="com.jkoolcloud.tnt4j.streams.inputs.CmdStream">
        <property name="HaltIfNoParser" value="false"/>

        <scenario name="Sample Win Cmd stream scenario">
            <step name="Step Windows">
                <request>typeperf "\Processor(_Total)\% Processor Time" -sc 1</request>
                <schedule-simple interval="25" units="Seconds" repeatCount="-1"/>
            </step>
        </scenario>

        <parser-ref name="ResponseParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that CmdStream referencing ResponseParser shall be used. Stream takes command response output string and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable entries.

Streaming scenario consists of one step defining system command to execute as request tag data and scheduler definition stating execute endlessly every 25 seconds.

ResponseParser parses command output string using Pattern property defined RegEx and produces activity snapshot containing field ProcessorTime.

Stream stops only when critical runtime error/exception occurs or application gets terminated.

*nix

This sample shows how to stream responses from executed *nix type OS command.

Sample files can be found in samples/unix-cmd-stream directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-ws.xsd">

    <parser name="ResponseParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityRegExParser">
        <property name="Pattern" value="(.*)"/>

        <field name="EventType" value="SNAPSHOT"/>
        <field name="TomcatActive" locator="1" locator-type="Index"/>
    </parser>

    <stream name="UnixCmdStream" class="com.jkoolcloud.tnt4j.streams.inputs.CmdStream">
        <property name="HaltIfNoParser" value="false"/>

        <scenario name="Sample *nix Cmd stream scenario">
            <step name="Step Is Active Tomcat">
                <request>systemctl is-active tomcat7.servicesystemctl is-active tomcat7.service</request>
                <schedule-simple interval="45" units="Seconds" repeatCount="-1"/>
            </step>
        </scenario>

        <parser-ref name="ResponseParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that CmdStream referencing ResponseParser shall be used. Stream takes command response output string and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable entries.

Streaming scenario consists of one step defining system command to execute as request tag data and scheduler definition stating execute endlessly every 45 seconds.

ResponseParser parses command output string using Pattern property defined RegEx and produces activity snapshot containing field TomcatActive.

Stream stops only when critical runtime error/exception occurs or application gets terminated.

JDBC

This sample shows how to stream JDBC provided ResultSet rows as activity events.

Sample files can be found in samples/b2bi-jdbc-stream directory (tnt4j-streams-ws module).

In run.bat/run.sh file set variable JDBC_LIBPATH value to reference JDBC implementation (PostgreSQL, MySQL, DB2, Oracle, etc.) libraries used by your environment.

See the sample at <tnt4j-streams>/tnt4j-streams-ws/samples/b2bi-jdbc-stream/tnt-data-source_oracle.xml.

Fill in these configuration value placeholders:

  • [HOST] — define your host for B2Bi database. Optionally you can completely change JDBC URL to match your JDBC driver, port and database.

  • [USER_NAME] — define your database user name.

  • [USER_PASS] — define your database user password.

Redirecting TNT4J Streams

This sample shows how to redirect tnt4j-stream-jmx (may be from multiple running instances) produced trackables to meshIQ over single TNT4J-Streams stream instance.

Sample files can be found in samples/stream-jmx directory.

To redirect tnt4j-stream-jmx (or any other TNT4J based producer) produced trackables, producer configuration file tnt4j.properties should contain such stanza:

event.sink.factory.EventSinkFactory: com.jkoolcloud.tnt4j.sink.impl.SocketEventSinkFactory
event.sink.factory.EventSinkFactory.LogSink: null
event.sink.factory.EventSinkFactory.Endpoint: IP_OF_STREAMS_RUNNING_MACHINE:9009
event.formatter: com.jkoolcloud.tnt4j.format.JSONFormatter

Change IP_OF_STREAMS_RUNNING_MACHINE to IP of machine running TNT4J-Streams RedirectTNT4JStream

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <java-object name="JMXRedirectOutput" class="com.jkoolcloud.tnt4j.streams.outputs.JKCloudJsonOutput"/>

    <stream name="SampleJMXRoutingStream" class="com.jkoolcloud.tnt4j.streams.inputs.RedirectTNT4JStream">
        <property name="RestartOnInputClose" value="true"/>
        <property name="Port" value="9009"/>

        <reference name="JMXRedirectOutput"/>

        <tnt4j-properties>
            <property name="event.formatter" value="com.jkoolcloud.tnt4j.streams.utils.RedirectTNT4JStreamFormatter"/>
        </tnt4j-properties>
    </stream>
</tnt-data-source>

Stream configuration states that SampleJMXRoutingStream referencing JMXRedirectOutput shall be used. Stream takes starts server socket on port defined by stream property Port. Stream listens server socket for inbound connections (accepts multiple). When connection gets accepted, stream reads incoming data from connection dedicated socket.

RestartOnInputClose property indicates that stream should initiate new instance of server socket if listened one gets closed or fails to accept inbound connection.

Stream referenced object JMXRedirectOutput sends JSON formatted data to meshIQ.

Stream also additionally sets one TNT4J framework property event.formatter. This allows us to use customized JSON formatter and avoid additional JSON reformatting in default TNT4J data flow.

You may also re-stream any TNT4J based producer logged trackables from file. Only requirement — trackables must be serialized in JSON format.

To do re-streaming from file, change sample configuration by replacing SampleJMXRoutingStream stream property Port to FileName referring file containing logged trackables in JSON format:

<stream name="SampleJMXRoutingStream" class="com.jkoolcloud.tnt4j.streams.inputs.RedirectTNT4JStream">
    <property name="FileName" value="tnt4j-stream-activities.log"/>

    <reference name="JMXRedirectOutput"/>

    <tnt4j-properties>
        <property name="event.formatter" value="com.jkoolcloud.tnt4j.streams.utils.RedirectTNT4JStreamFormatter"/>
    </tnt4j-properties>
</stream>

MS Excel Document

Rows

This sample shows how to stream MS Excel workbook rows as activity events.

Sample files can be found in samples/xlsx-rows directory.

Records in this file are from year 2010, e.g. 12 Jul 2010, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="ExcelRowParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityExcelRowParser">
        <field name="StartTime" locator="A" locator-type="Label" format="dd MMM yyyy HH:mm:ss" locale="en-US"/>
        <field name="ServerIp" locator="B" locator-type="Label"/>
        <field name="ApplName" value="orders"/>
        <field name="Correlator" locator="C" locator-type="Label"/>
        <field name="UserName" locator="D" locator-type="Label"/>
        <field name="EventName" locator="E" locator-type="Label"/>
        <field name="EventType" locator="E" locator-type="Label">
            <field-map source="Order Placed" target="START"/>
            <field-map source="Order Received" target="RECEIVE"/>
            <field-map source="Order Processing" target="OPEN"/>
            <field-map source="Order Processed" target="SEND"/>
            <field-map source="Order Shipped" target="END"/>
        </field>
        <field name="MsgValue" locator="H" locator-type="Label"/>
    </parser>

    <stream name="SampleExcelRowsStream" class="com.jkoolcloud.tnt4j.streams.inputs.ExcelRowStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="./tnt4j-streams-msoffice/samples/xlsx-rows/sample.xlsx"/>
        <property name="RangeToStream" value="1:"/>
        <property name="SheetsToProcess" value="Sheet*"/>

        <parser-ref name="ExcelRowParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that SampleExcelRowsStream referencing ExcelRowsParser shall be used. Stream takes workbook sheet row and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable rows.

SampleExcelRowsStream reads data from ./tnt4j-streams-msoffice/samples/xlsx-rows/sample.xlsx file.

RangeToStream defines range of rows to be streamed from each matching sheet — from first row to the end.

SheetsToProcess property defines sheet name filtering mask using wildcard string. It is also allowed to use RegEx like Sheet(1|3|5) (in this case just sheets with names Sheet1, Sheet3 and Sheet5 will be processed).

ExcelRowParser parser uses literal sheet column indicators as locators (e.g., A, D, AB).

StartTime fields defines format and locale to correctly parse field data string. EventType uses manual field string mapping to TNT4J event field value.

ExcelRowStream uses DOM based MS Excel file reading, thus memory consumption for large file may be significant, but it allows random cells access and precise formula evaluation. In case memory consumption is critical factor, use ExcelSXSSFRowStream instead of ExcelRowStream. It uses Apache POI SXSSF API to read MS Excel as a stream consistently iterating over workbook sheets rows and cells. Thus, it may have some drawback on cell formula evaluation. For more information see Apache POI spreadsheet documentation. 

Sheets

This sample shows how to stream MS Excel workbook sheets as activity events.

Sample files can be found in samples/xlsx-sheets directory.

Records in this file are from year 2010, e.g. 12 Jul 2010, so when sending the events data to meshIQ, please do not forget to adjust the dashboard time frame to that period!

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="ExcelSheetParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityExcelSheetParser">
        <field name="StartTime" locator="B2" locator-type="Label" format="dd MMM yyyy HH:mm:ss" locale="en-US"/>
        <field name="ServerIp" locator="B3" locator-type="Label"/>
        <field name="ApplName" value="orders"/>
        <field name="Correlator" locator="B4" locator-type="Label"/>
        <field name="UserName" locator="B5" locator-type="Label"/>
        <field name="EventName" locator="B6" locator-type="Label"/>
        <field name="EventType" locator="B6" locator-type="Label">
            <field-map source="Order Placed" target="START"/>
            <field-map source="Order Received" target="RECEIVE"/>
            <field-map source="Order Processing" target="OPEN"/>
            <field-map source="Order Processed" target="SEND"/>
            <field-map source="Order Shipped" target="END"/>
        </field>
        <field name="MsgValue" locator="B9" locator-type="Label"/>
    </parser>

    <stream name="SampleExcelSheetsStream" class="com.jkoolcloud.tnt4j.streams.inputs.ExcelSheetStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="./tnt4j-streams-msoffice/samples/xlsx-sheets/sample.xlsx"/>
        <property name="SheetsToProcess" value="Sheet*"/>

        <parser-ref name="ExcelSheetParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that SampleExcelSheetsStream referencing ExcelSheetParser shall be used. Stream takes workbook sheet and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable sheets.

SampleExcelRowsStream reads data from ./tnt4j-streams-msoffice/samples/xlsx-sheets/sample.xlsx file.

SheetsToProcess property defines sheet name filtering mask using wildcard string. It is also allowed to use RegEx like Sheet(1|3|5) (in this case just sheets with names Sheet1, Sheet3 and Sheet5 will be processed).

ExcelSheetParser parser uses literal sheet cell indicators as locators (e.g., A1, D5, AB12 where letters identifies column and number identifies row).

StartTime fields defines format and locale to correctly parse field data string. EventType uses manual field string mapping to TNT4J event field value.

Collected Performance Metrics Streaming

This sample shows how to stream collectd monitoring reports data as activity events.

Sample files can be found in samples/collectd-json directory (tnt4j-streams-core module).

To use this sample collectd should be running Write HTTP plugin.

Sample report data is available in stats.json file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="CollectdStatsDataParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="EventType" value="SNAPSHOT"/>
        <field name="EventName" locator="type|type_instance" locator-type="Label" separator=" "/>
        <field name="Category" locator="plugin|plugin_instance" locator-type="Label" separator=" "/>
        <field name="ServerName" locator="host" locator-type="Label"/>

        <field name="${FieldNameLoc}" locator="values" locator-type="Label" value-type="${ValueTypeLoc}" split="true">
            <field-locator id="FieldNameLoc" locator="dsnames" locator-type="Label"/>
            <field-locator id="ValueTypeLoc" locator="dstypes" locator-type="Label"/>
        </field>

        <field name="StartTime" locator="time" locator-type="Label" datatype="Timestamp" units="Seconds"/>
    </parser>

    <parser name="CollectdReqBodyParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="MsgBody" locator="$" locator-type="Label" transparent="true" split="true">
            <parser-ref name="CollectdStatsDataParser" aggregation="Relate"/>
        </field>
        <field name="EventType" value="Activity"/>
        <field name="ApplName" value="collectd"/>
    </parser>

    <parser name="CollectdReqParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="Transport" locator="ActivityTransport" locator-type="Label"/>
        <field name="MsgBody" locator="ActivityData" locator-type="Label">
            <parser-ref name="CollectdReqBodyParser"/>
        </field>
    </parser>

    <stream name="CollectdStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="9595"/>

        <parser-ref name="CollectdReqParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that HttpStream referencing CollectdReqParser shall be used. Stream takes HTTP request received from collectd server and passes it to parser.

HttpStream starts HTTP server on port defined using Port property. HaltIfNoParser property indicates that stream should skip unparseable entries. Stream puts received request payload data as byte[] to map using key ActivityData.

CollectdReqParser by default converts byte[] for entry ActivityData to string and uses stacked parser named CollectdReqBodyParser to parse HTTP request contained JSON format data.

CollectdReqBodyParser maps collectd report JSON data to activity (field EventType) containing set of snapshots (field MsgBody) carrying system metrics data. Each snapshot is parsed using stacked CollectdStatsDataParser parser (map parser because parent JSON parser already made map data structures from Raw collectd report JSON data).

CollectdStatsDataParser maps map entries to snapshot fields EventName, Category, ServerName, StartTime. Also, this parser uses dynamic field named ${FieldNameLoc}. Attribute values containing variable expressions ${} indicates that these attributes can have dynamic values resolved from streamed data. Such variable expressions has to reference locators identifiers (field-locator attribute id) to resolve and fill actual data. Field attribute split, that if field value locator resolves collection (list or array) of values, values of that collection should be split into separate fields of produced activity.

Nagios Reports Streaming

This sample shows how to stream Nagios monitoring reports data as activity events.

Sample files can be found in samples/nagios-nagios2json directory (tnt4j-streams-ws module).

To use this sample Nagios should be running extension nagios2json. See Configure Nagios on Linux for details. nagios2json extension works as CGI script, but can be handled as simple RESTful service.

Sample report data is available in report.json file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-ws.xsd">

    <parser name="SnapshotParser"
            class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">

        <field name="EventType" value="Snapshot"/>
        <field name="ApplName" value="Nagios"/>
        <field name="EventName" locator="service" locator-type="Label"/>
        <field name="Status" locator="status" locator-type="Label" value-type="enum"/>
        <field name="Message" locator="plugin_output" locator-type="Label"/>
        <field name="Category" locator="hostname" locator-type="Label"/>
        <field name="Duration" locator="duration" locator-type="Label" value-type="age"/>
        <field name="StartTime" locator="last_state_change" locator-type="Label" datatype="Timestamp" units="Seconds"/>
    </parser>

    <parser name="ResponseParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="MsgBody" locator="$.data" locator-type="Label" transparent="true" split="true">
            <parser-ref name="SnapshotParser" aggregation="Relate"/>
        </field>

        <field name="EventType" value="Activity"/>
        <field name="StartTime" locator="$.created" locator-type="Label" datatype="Timestamp" units="Seconds"/>
        <field name="ApplName" value="nagios2json"/>
        <field name="Message" separator=", ">
            <field-locator locator="$.version" locator-type="Label"/>
            <field-locator locator="$.running" locator-type="Label"/>
            <field-locator locator="$.servertime" locator-type="Label" datatype="Timestamp" units="Seconds"/>
            <field-locator locator="$.localtime" locator-type="Label"/>
        </field>
    </parser>

    <stream name="RESTfulSampleNagiosStream" class="com.jkoolcloud.tnt4j.streams.inputs.RestStream">
        <property name="HaltIfNoParser" value="false"/>

        <scenario name="Nagios2JSON stream scenario">
            <step name="Step 1"
                  url="http://[YOUR_NAGIOS_SERVER_IP]/nagios/cgi-bin/nagios2json.cgi?servicestatustypes=31"
                  method="GET"
                  username="myNagiosUserName"
                  password="myNagiosUserSecretPassword">
                <schedule-cron expression="0/15 * * * * ? *"/>
            </step>
        </scenario>

        <parser-ref name="ResponseParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that RestStream referencing ResponseParser shall be used. Stream takes response received JSON string and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable entries.

Streaming scenario step defines:

  • Nagios service request URL and nagios2json parameters (e.g., servicestatustypes=31)

  • request method GET to comply cgi.

  • Nagios service user credentials - user name and password

  • Scheduler is set to Cron expression every 15 seconds.

You may also add additional steps to retrieve different reports defining different nagios2json parameters.

Nagios sends report as activity wrapping multiple metrics (snapshots).

ResponseParser maps Nagios report JSON data to activity (field EventType) containing set of snapshots (field MsgBody) carrying system state/metrics data. Each snapshot is parsed using stacked SnapshotParser parser (map parser because parent JSON parser already made map data structures from Raw Nagios report JSON data).

SnapshotParser maps map entries to snapshot fields ApplName, EventName, Status, Message, Category, Duration and StartTime. Status and Duration fields also defines value types: Status is enum, Duration is age.

IBM MQ Error Log Streaming

This sample shows how to stream IBM MQ error log entries as activity events.

Sample files can be found in samples/ibm-mq-err-log directory (tnt4j-streams-core module).

Sample error log file is available in AMQERR01.LOG file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="MQErrLogParser" class="com.jkoolcloud.tnt4j.streams.custom.parsers.IBMMQLogParser">
        <property name="UseActivityDataAsMessageForUnset" value="true"/>
        <property name="ActivityDelim" value="----[\n\r]+"/>

        <field name="EventType" value="EVENT"/>
        <field name="ResourceName" locator="FileName" locator-type="StreamProp"/>

        <field name="StartTimeCommon" separator=" " format="M/d/yyyy HH:mm:ss" datatype="DateTime" transparent="true">
            <field-locator locator="Date" locator-type="Label"/>
            <field-locator locator="Time" locator-type="Label"/>
        </field>
        <field name="Process" locator="Process" locator-type="Label"/>
        <field name="ProcessId" locator="pid" locator-type="Label"/>
        <field name="ThreadId" locator="tid" locator-type="Label"/>
        <field name="UserName" locator="User" locator-type="Label"/>
        <field name="ApplName" locator="Program" locator-type="Label"/>
        <field name="ServerName" locator="Host" locator-type="Label"/>
        <field name="Location" locator="Installation" locator-type="Label"/>
        <field name="VRMF" locator="VRMF" locator-type="Label"/>
        <field name="QMGR" locator="QMgr" locator-type="Label"/>
        <field name="EventName" locator="ErrCode" locator-type="Label"/>
        <field name="Exception" locator="ErrText" locator-type="Label"/>
        <field name="Explanation" locator="Explanation" locator-type="Label"/>
        <field name="Action" locator="Action" locator-type="Label"/>
        <field name="Where" locator="Where" locator-type="Label"/>

        <!-- IBM MQ 9.1 additional attributes -->
        <field name="Severity" locator="Severity" locator-type="Label"/>
        <field name="StartTimeUTC" locator="TimeUTC" locator-type="Label" format="yyyy-MM-dd'T'HH:mm:ss.SSSX" datatype="DateTime"
               transparent="true"/>

        <!--        <field name="CommentInsert1" locator="CommentInsert1" locator-type="Label"/>-->
        <!--        <field name="CommentInsert1" locator="CommentInsert2" locator-type="Label"/>-->
        <!--        <field name="CommentInsert1" locator="CommentInsert3" locator-type="Label"/>-->

        <!--        <field name="ArithInsert1" locator="ArithInsert3" locator-type="Label"/>-->
        <!--        <field name="ArithInsert1" locator="ArithInsert3" locator-type="Label"/>-->
        <!--        <field name="ArithInsert1" locator="ArithInsert3" locator-type="Label"/>-->

        <field name="StartTime" value="">
            <field-transform lang="groovy">
                ${StartTimeUTC} == null ? ${StartTimeCommon} : ${StartTimeUTC}
            </field-transform>
        </field>

        <field name="AllRestLogEntryValues" locator="#" locator-type="Label"/>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="./tnt4j-streams-core/samples/ibm-mq-err-log/AMQERR*.LOG"/>
        <property name="RestoreState" value="false"/>
        <property name="FilePolling" value="true"/>
        <property name="FileReadDelay" value="20"/>
        <property name="StartFromLatest" value="false"/>
        <property name="ActivityDelim" value="-----"/>
        <property name="KeepLineSeparators" value="true"/>

        <parser-ref name="MQErrLogParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that FileStream referencing MQErrLogParser shall be used. Stream reads IBM MQ error log entries from ./tnt4j-streams-core/samples/ibm-mq-err-log/AMQERR01.LOG file contents and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable entries.

MQErrLogParser maps IBM MQ error log entry resolved fields to activity event fields. There is additional field ResourceName resolved from stream configuration data and referring stream input (IBM MQ error log) file name.

JSON format IBM MQ Error Log Streaming

This sample shows how to stream JSON formatted IBM MQ (version starting 9.0.5) error log entries as activity events.

Sample files can be found in samples/ibm-mq-err-log directory (tnt4j-streams-core module).

Sample error log file is available in AMQERR01.JSON file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="MQErrLogJSONParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="UseActivityDataAsMessageForUnset" value="true"/>

        <field name="EventType" value="EVENT"/>
        <field name="ResourceName" locator="FileName" locator-type="StreamProp"/>

        <field name="StartTime" locator="$.ibm_datetime" format="yyyy-MM-dd'T'HH:mm:ss.SSSX" locator-type="Label"/>
        <field name="Process" locator="$.ibm_processName" locator-type="Label"/>
        <field name="ProcessId" locator="$.ibm_processId" locator-type="Label"/>
        <field name="ThreadId" locator="$.ibm_threadId" locator-type="Label"/>
        <field name="UserName" locator="$.ibm_userName" locator-type="Label"/>
        <field name="ServerName" locator="$.ibm_serverName" locator-type="Label"/>
        <field name="Location" locator="$.ibm_installationName" locator-type="Label"/>
        <field name="VRMF" locator="$.ibm_version" locator-type="Label"/>
        <field name="QMGR" locator="$.ibm_qmgrId" locator-type="Label"/>
        <field name="EventName" locator="$.ibm_messageId" locator-type="Label"/>
        <field name="Exception" locator="$.message" locator-type="Label"/>
        <field name="Where" locator="$.module" locator-type="Label"/>

        <field name="MsgVariable1" locator="$.ibm_arithInsert1" locator-type="Label"/>
        <field name="MsgVariable2" locator="$.ibm_arithInsert2" locator-type="Label"/>
        <field name="MsgComment1" locator="$.ibm_commentInsert1" locator-type="Label"/>
        <field name="MsgComment2" locator="$.ibm_commentInsert2" locator-type="Label"/>
        <field name="MsgComment3" locator="$.ibm_commentInsert3" locator-type="Label"/>
        <field name="Severity" locator="$.loglevel" locator-type="Label"/> <!--  INFO, WARNING, or ERROR -->
        <field name="Sequence" locator="$.ibm_sequence" locator-type="Label"/>
        <field name="RemoteHost" locator="$.ibm_remoteHost" locator-type="Label"/>
        <field name="Host" locator="$.host" locator-type="Label"/>
        <field name="InstallationDir" locator="$.ibm_installationDir" locator-type="Label"/>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="./tnt4j-streams-core/samples/ibm-mq-err-log/AMQERR*.JSON"/>
        <property name="RestoreState" value="false"/>
        <property name="FilePolling" value="true"/>
        <property name="FileReadDelay" value="20"/>
        <property name="StartFromLatest" value="false"/>
        <property name="ActivityDelim" value="EOL"/>

        <parser-ref name="MQErrLogJSONParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that FileStream referencing MQErrLogJSONParser shall be used. Stream reads JSON formatted IBM MQ error log entries (one per line) from ./tnt4j-streams-core/samples/ibm-mq-err-log/AMQERR01.JSON file contents and passes it to parser.

HaltIfNoParser property indicates that stream should skip unparseable entries.

MQErrLogJSONParser maps IBM MQ error JSON log entry resolved fields to activity event fields. There is additional field ResourceName resolved from stream configuration data and referring stream input (IBM MQ error log) file name.

JSON formatted IBM MQ error log entries does not have some fields e.g. Action, Explanation, like ordinary log has.

See JSON format diagnostic messages for more details.

String Ranges Streaming

This sample shows how to stream SOCGEN messages strings from file entries as activity events. Each file entry string represents activity event and fields of event are substring ranges.

Sample files can be found in samples/string-ranges directory (tnt4j-streams-core module).

Sample message data is available in strings.txt file. Now it contains only one entry.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="StrRangesParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityStringParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="EventType" value="EVENT"/>
        <field name="EventName" value="SOCGEN_Msg_Data"/>

        <field name="TransID">  <!-- offsets 0-13 inclusive, length=14, expected value: TID_TEST3B_456 -->
            <field-locator locator="0:14" locator-type="Range"/>
        </field>
        <field name="TransType">  <!-- offsets 20-35 inclusive, length=16, expected value: TYPE_TEST3B_SALE -->
            <field-locator locator="20:36" locator-type="Range"/>
        </field>
        <field name="TransValue">  <!-- offsets 70-89 inclusive, field length=20, expected value: AMT_TEST3B_USD123.45  -->
            <field-locator locator="70:90" locator-type="Range"/>
        </field>
        <field name="UserData">  <!-- offsets 123-145 inclusive, length=23, expected value: TEST3B_Model iSeries123 -->
            <field-locator locator="123:146" locator-type="Range"/>
        </field>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="FileName" value="./tnt4j-streams-core/samples/string-ranges/strings.txt"/>

        <parser-ref name="StrRangesParser"/>
    </stream>
</tnt-data-source>

The stream configuration specifies that a FileStream referencing StrRangesParser must be used. The stream reads message entries from the file ./tnt4j-streams-core/samples/string-ranges/strings.txt and passes them to the parser.

The HaltIfNoParser property indicates that the stream should skip unparseable entries.

StrRangesParser extracts substrings based on defined ranges and maps them to activity event fields. For example, TransID is defined as the substring ranging from index 0 to 13 within the provided string.

The lower bound of the range is inclusive, while the upper bound is exclusive.

Range values can be defined as follows:

  • :x – from the beginning of the string to the character at index x (exclusive).

  • x:y – from the character at index x (inclusive) to the character at index y (exclusive).

  • x: – from the character at index x (inclusive) to the end of the string.

IBM MQ RFH2/JMS Streaming

This sample shows how to stream IBM MQ error log entries as activity events.

Sample files can be found in samples/ibm-mq-err-log directory (tnt4j-streams-core module).

Sample error log file is available in AMQERR01.LOG file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="RFH2FoldersParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
        <field name="PayloadDataType" locator="/rfh2Folders/mcd/Msd" locator-type="Label"/>
        <field name="DestinationQ" locator="/rfh2Folders/jms/Dst" locator-type="Label"/>
        <field name="StartTime" locator="/rfh2Folders/jms/Tms" locator-type="Label" datatype="Timestamp" units="Milliseconds"/>
    </parser>

    <parser name="JMSPayloadParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="UserData" locator="*" locator-type="Label"/>
    </parser>

    <parser name="RFH2Parser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityRFH2Parser">
        <property name="TranslateNumValues" value="true"/>
        <property name="UseActivityDataAsMessageForUnset" value="false"/>

        <field name="EventType" value="EVENT"/>
        <field name="EventName" value="IBM MQ RFH2/JMS Payload"/>
        <field name="FoldersData" locator="rfh2Folders" locator-type="Label" transparent="true">
            <parser-ref name="RFH2FoldersParser" aggregation="Merge"/>
        </field>

        <field name="JMS_Payload" locator="jmsMsgPayload" locator-type="Label" transparent="true">
            <parser-ref name="JMSPayloadParser" aggregation="Merge"/>
        </field>
    </parser>

    <stream name="RFH2JMSFileStream" class="com.jkoolcloud.tnt4j.streams.inputs.BytesInputStream">
        <property name="FileName" value="./tnt4j-streams-wmq/samples/rfh2_jms/rfh2_jms.bin"/>
        <property name="RestoreState" value="false"/>

        <parser-ref name="RFH2Parser"/>
    </stream>
</tnt-data-source>

The stream configuration specifies that an RFH2JMSFileStream referencing RFH2Parser must be used. The stream reads message entries from the file ./tnt4j-streams-wmq/samples/rfh2_jms/rfh2_jms.bin and passes them to the parser. Since ActivityRFH2Parser is an extension of the map parser, it produces a map with two predefined keys:

  • rfh2Folders – RFH2 folders data as an XML string.

  • jmsMsgPayload – JMS message payload as a deserialized object, or as raw bytes if deserialization cannot be performed.

RFH2Parser resolves the RFH2 folders and JMS message payload into the predefined fields rfh2Folders and jmsMsgPayload. These field values are then passed to RFH2FoldersParser and JMSPayloadParser for further parsing into fields of the EVENT named IBM_MQ_RFH2/JMS_PAYLOAD.

  • RFH2FoldersParser uses XPath expressions to extract values.

  • JMSPayloadParser copies all JMS MapMessage entries as fields of the EVENT named IBM_MQ_RFH2/JMS_PAYLOAD.

JSR-203 FileSystem

SFTP File Feed

This sample shows how to stream scp: file system (over SSH) provided file characters feed data as activity events.

Sample files can be found in <tnt4j-streams>/tnt4j-streams-fs/samples/sftp-file-feed/tnt-data-source.xml.

SCP(SSH) File Feed

This sample shows how to stream scp: file system (over SSH) provided file characters feed data as activity events.

Sample files can be found in <tnt4j-streams>/tnt4j-streams-fs/samples/ssh-file-feed/tnt-data-source.xml.

SCP(SSH) File Lines

This sample shows how to stream scp: file system (over SSH) provided file lines as activity events.

Sample files can be found in <tnt4j-streams>/tnt4j-streams-fs/samples/ssh-file-lines/tnt-data-source.xml.

Zip File Lines

This sample shows how to stream zip: file system provided file lines as activity events.

Sample files can be found in <tnt4j-streams>/tnt4j-streams-fs/samples/zip-fs-file-lines/tnt-data-source.xml.

Integrating TNT4J-Streams into Custom API

See TNT4J-Streams API Use Samples of tnt4j-streams-samples module.