Skip to main content
Top
Review meshIQ Products (G2.com)

Streaming Data to TNT4J-Streams: HTTP, IBM MQ, and JS Examples

This guide shows how to stream data into TNT4J-Streams from common sources—HTTP (file, form, AJAX), IBM MQ (Broker Events, PCF, Trace), Java (JAX-WS/JAX-RS), and JavaScript/Node.js—using parsers and stacked-parser patterns to turn raw payloads into actionable activity events.

HTTP Request File

This sample shows how to stream activity events received over an HTTP request as a file. The sample also shows how to use the stacked parsers technique to extract message payload data.

The sample files can be found in the samples/http-file directory.

Over HTTP sent sample file is log.txt - a snapshot of an Apache access log depicting some HTTP server activity.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
    </parser>

    <parser name="SampleHttpReqParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="Transport" locator="ActivityTransport" locator-type="Label"/>
        <field name="MsgBody" locator="ActivityData" locator-type="Label">
            <parser-ref name="AccessLogParserCommon"/>
        </field>
    </parser>

    <stream name="SampleHttpStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="8080"/>
        <!--<property name="UseSSL" value="true"/>-->
        <!--<property name="Keystore" value="path_to_keystore_file"/>-->
        <!--<property name="KeystorePass" value="somePassword"/>-->
        <!--<property name="KeyPass" value="somePassword"/>-->
        <parser-ref name="SampleHttpReqParser"/>
    </stream>
</tnt-data-source>

The stream configuration states that HttpStream, referencing SampleHttpReqParser, should be used.

HttpStream starts an HTTP server on the port defined by the Port property. The HaltIfNoParser property indicates that the stream should skip unparsable entries. The stream puts received request payload data as a byte[] into the map using the key ActivityData.

SampleHttpReqParser, by default converts byte[] in entry ActivityData to a string and uses the stacked parser named AccessLogParserCommon to parse the format.

AccessLogParserCommon is the same as in the Apache Access log single file sample, so refer to it for more details.

To parse other data instead of an Apache Access Log, replace AccessLogParserCommon with a parser that complies with your data format.

The stream stops only when a critical runtime error/exception occurs, or when the application is terminated.

HTTP Request Form

This sample shows how to the stream activity events received over an HTTP request as Form Data.

Sample files can be found in the samples/http-form directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="SampleFormDataParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="Location" locator="clientip" locator-type="Label"/>
        <field name="UserName" locator="auth" locator-type="Label"/>
        <field name="StartTime" locator="timestamp" locator-type="Label" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="verb" locator-type="Label"/>
        <field name="ResourceName" locator="request" locator-type="Label"/>
        <field name="CompCode" locator="response" locator-type="Label">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="response" locator-type="Label"/>
        <field name="MsgValue" locator="bytes" locator-type="Label"/>
    </parser>

    <stream name="SampleHttpStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="8080"/>
        <!--<property name="UseSSL" value="true"/>-->
        <!--<property name="Keystore" value="path_to_keystore_file"/>-->
        <!--<property name="KeystorePass" value="somePassword"/>-->
        <!--<property name="KeyPass" value="somePassword"/>-->
        <parser-ref name="SampleFormDataParser"/>
    </stream>
</tnt-data-source>

The stream configuration states that HttpStream, referencing SampleFormDataParser, should be used.

HttpStream starts an HTTP server on the port defined using the Port property. The HaltIfNoParser property indicates that the stream should skip unparsable entries. The stream puts received form parameter data  into a map and passes it to the parser.

SampleFormDataParser performs form data mapping to TNT4J activity event data using form data parameter name labels.

The stream stops only when a critical runtime error/exception occurs, or when the application is terminated.

WMQ Message Broker

This sample shows how to stream activity events received over WMQ as Message Broker event messages.

Sample files can be found in the samples/message-broker directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <java-object name="MQMsgToStringPreParser" class="com.jkoolcloud.tnt4j.streams.preparsers.MQMessageToStringPreParser"/>

    <parser name="EventParser" class="com.jkoolcloud.tnt4j.streams.parsers.MessageActivityXmlParser">
        <property name="Namespace"
                  value="wmb=http://www.ibm.com/xmlns/prod/websphere/messagebroker/6.1.0/monitoring/event"/>

        <reference name="MQMsgToStringPreParser"/>

        <!--field name="ServerName" value="host-name-for-broker"/--> <!-- defaults to host name where jKool LLC TNT4J-Streams is running -->
        <!--field name="ServerName" locator="/wmb:event/wmb:eventPointData/wmb:messageFlowData/wmb:broker/@wmb:hostName" locator-type="Label"/--> <!-- when broker supports this -->
        <!--field name="ServerIp" locator="/wmb:event/wmb:eventPointData/ServerIp" locator-type="Label"/-->
        <field name="ApplName"
               locator="/wmb:event/wmb:eventPointData/wmb:messageFlowData/wmb:messageFlow/@wmb:uniqueFlowName"
               locator-type="Label"/>
        <field name="ResourceName" locator="/wmb:event/wmb:eventPointData/wmb:messageFlowData/wmb:node/@wmb:nodeLabel"
               locator-type="Label"/>

        <field name="RawMessage" locator="/wmb:event/wmb:bitstreamData/wmb:bitstream" locator-type="Label" datatype="String"
               transparent="true"/>
        <field name="RawMessageEncoding" locator="/wmb:event/wmb:bitstreamData/wmb:bitstream/@wmb:encoding" locator-type="Label"
               datatype="String" transparent="true"/>

        <field name="Message" locator="RawMessage" locator-type="Activity">
            <field-transform lang="groovy" phase="raw"><![CDATA[
                "base64Binary".equals(${RawMessageEncoding})
                    ? Utils.base64Decode($fieldValue, "UTF-8")
                    : "hexBinary".equals(${RawMessageEncoding})
                        ? Utils.getString(Utils.decodeHex($fieldValue), "UTF-8")
                        : $fieldValue
            ]]></field-transform>
        </field>

        <field name="EventName" locator="/wmb:event/wmb:eventPointData/wmb:eventData/wmb:eventIdentity/@wmb:eventName"
               locator-type="Label"/>
        <field name="EventType" locator="/wmb:event/wmb:eventPointData/wmb:messageFlowData/wmb:node/@wmb:nodeType"
               locator-type="Label">
            <field-map source="ComIbmMQInputNode" target="RECEIVE"/>
            <field-map source="ComIbmMQOutputNode" target="SEND"/>
            <field-map source="ComIbmMQGetNode" target="RECEIVE"/>
            <field-map source="ComIbmJMSClientInputNode" target="RECEIVE"/>
            <field-map source="ComIbmJMSClientOutputNode" target="SEND"/>
            <field-map source="ComIbmJMSClientReplyNode" target="SEND"/>
            <field-map source="ComIbmJMSClientReceive" target="RECEIVE"/>
            <field-map source="ComIbmJMSHeader.msgnode" target="RECEIVE"/>
            <field-map source="ComIbmHTTPAsyncRequest" target="RECEIVE"/>
            <field-map source="ComIbmHTTPAsyncResponse" target="SEND"/>
            <field-map source="ComIbmHTTPHeader" target="RECEIVE"/>
            <field-map source="ComIbmWSInputNode" target="RECEIVE"/>
            <field-map source="ComIbmWSReplyNode" target="SEND"/>
            <field-map source="ComIbmWSRequestNode" target="RECEIVE"/>
            <field-map source="ComIbmSOAPInputNode" target="RECEIVE"/>
            <field-map source="ComIbmSOAPReplyNode" target="SEND"/>
            <field-map source="ComIbmSOAPRequestNode" target="RECEIVE"/>
            <field-map source="ComIbmSOAPAsyncRequestNode" target="RECEIVE"/>
            <field-map source="ComIbmSOAPAsyncResponseNode" target="SEND"/>
            <field-map source="ComIbmSOAPWrapperNode" target="CALL"/>
            <field-map source="ComIbmSOAPExtractNode" target="CALL"/>
            <field-map source="SRRetrieveEntityNode" target="CALL"/>
            <field-map source="SRRetrieveITServiceNode" target="CALL"/>
            <field-map source="ComIbmDatabaseInputNode" target="RECEIVE"/>
            <field-map source="ComIbmDatabaseNode" target="CALL"/>
            <field-map source="ComIbmDatabaseRetrieveNode" target="RECEIVE"/>
            <field-map source="ComIbmDatabaseRouteNode" target="SEND"/>
            <field-map source="ComIbmFileInputNode" target="RECEIVE"/>
            <field-map source="ComIbmFileReadNode" target="CALL"/>
            <field-map source="ComIbmFileOutputNode" target="SEND"/>
            <field-map source="ComIbmFTEInputNode" target="RECEIVE"/>
            <field-map source="ComIbmFTEOutputNode" target="SEND"/>
            <field-map source="ComIbmTCPIPClientInputNode" target="RECEIVE"/>
            <field-map source="ComIbmTCPIPClientOutputNode" target="SEND"/>
            <field-map source="ComIbmTCPIPClientRequestNode" target="RECEIVE"/>
            <field-map source="ComIbmTCPIPServerInputNode" target="RECEIVE"/>
            <field-map source="ComIbmTCPIPServerOutputNode" target="SEND"/>
            <field-map source="ComIbmTCPIPServerRequestNode" target="RECEIVE"/>
            <field-map source="ComIbmCORBARequestNode" target="RECEIVE"/>
            <field-map source="" target="CALL"/>
        </field>
        <field name="Correlator"
               locator="/wmb:event/wmb:eventPointData/wmb:eventData/wmb:eventCorrelation/@wmb:localTransactionId"
               locator-type="Label"/>
        <field name="ElapsedTime" value="0" datatype="Number"/>
        <field name="EndTime" locator="/wmb:event/wmb:eventPointData/wmb:eventData/wmb:eventSequence/@wmb:creationTime"
               locator-type="Label"
               datatype="DateTime" format="yyyy-MM-dd'T'HH:mm:ss.SSSSSSX"/>
        <!--field name="ReasonCode" locator="/wmb:event/wmb:eventPointData/ReasonCode" locator-type="Label" datatype="Number"/-->
        <!-- *** Use following signature definition for WMQ messages ***
        <field name="TrackingId" separator="#!#" value-type="signature">
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='MsgType']/@wmb:value" locator-type="Label" datatype="Number"/>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='Format']/@wmb:value" locator-type="Label"/>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='MsgId']/@wmb:value" locator-type="Label" datatype="Binary" format="hexBinary"/>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='UserIdentifier']/@wmb:value" locator-type="Label">
              <field-transform name="UserIdLowerCase" lang="groovy">
                  StringUtils.lowerCase($fieldValue)
              </field-transform>
          </field-locator>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='PutApplType']/@wmb:value" locator-type="Label"/>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='PutApplName']/@wmb:value" locator-type="Label"/>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='PutDate']/@wmb:value" locator-type="Label"/>
          <field-locator locator="/wmb:event/wmb:applicationData/wmb:simpleContent[@wmb:name='PutTime']/@wmb:value" locator-type="Label"/>
        </field>
        -->
        <!--field name="StartTime" locator="/wmb:event/wmb:eventPointData/wmb:eventData/wmb:eventSequence/@wmb:creationTime" 
            locator-type="Label" datatype="DateTime" format="yyyy-MM-dd'T'HH:mm:ss.SSSSSSX"/-->
        <field name="CompCode" locator="/wmb:event/wmb:eventPointData/wmb:eventData/wmb:eventIdentity/@wmb:eventName"
               locator-type="Label">
            <field-map source="FlowRollback" target="ERROR"/>
            <field-map source="" target="SUCCESS"/>
        </field>
        <!--field name="Tag" locator="/wmb:event/wmb:eventPointData/Tag" locator-type="Label"/-->
        <!--field name="UserName" locator="/wmb:event/wmb:eventPointData/UserName" locator-type="Label"/-->
    </parser>

    <stream name="EventStream" class="com.jkoolcloud.tnt4j.streams.inputs.WmqStream">
        <property name="QueueManager" value="QMGR"/>
        <property name="Queue" value="EVENT.QUEUE"/>
        <parser-ref name="EventParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that WmqStream referencing EventParser shall be used. The stream deserializes the message to string and passes it to the parser.

QueueManager property defines name of queue manager and Queue property defines name of queue to get messages.

EventParser is of type MessageActivityXmlParser meaning that it will parse messages de-serialized into XML strings.

Namespace property adds wmb namespace definition mapping to mapping http://www.ibm.com/xmlns/prod/websphere/messagebroker/6.1.0/monitoring/event.

WMQ PCF Messages Streaming

This sample shows how to stream activity events received over WMQ as PCF messages.

Sample files can be found in the samples/pcf-message directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-wmq_pcf.xsd">

    <parser name="PCFEventsParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityPCFParser">
        <property name="TranslateNumValues" value="true"/>

        <field name="EventType" value="EVENT"/>

        <!-- header fields -->
        <field name="Command" locator="MQCFH.Command" locator-type="Label"/>
        <field name="MsgSeqNumber" locator="MQCFH.MsgSeqNumber" locator-type="Label"/>
        <field name="Control" locator="MQCFH.Control" locator-type="Label"/>
        <field name="CompCode" locator="MQCFH.CompCode" locator-type="Label">
            <field-map source="0" target="SUCCESS"/>
            <field-map source="1" target="WARNING"/>
            <field-map source="MQCC_OK" target="SUCCESS"/>
            <field-map source="MQCC_WARNING" target="WARNING"/>
            <field-map source="" target="ERROR"/>
        </field>
        <field name="ReasonCode" locator="MQCFH.Reason" locator-type="Label" datatype="Number"/>
        <field name="ParameterCount" locator="MQCFH.ParameterCount" locator-type="Label"/>

        <!-- message fields -->
        <field name="QMgrName" locator="MQCA_Q_MGR_NAME" locator-type="Label"/>
        <field name="ServerName" locator="MQCACF_HOST_NAME" locator-type="Label"/>
        <field name="StartTime" separator=" " datatype="DateTime" format="yyyy-MM-dd HH:mm:ss">
            <field-locator locator="MQCAMO_START_DATE" locator-type="Label"/>
            <field-locator locator="MQCAMO_START_TIME" locator-type="Label"/>
        </field>
        <field name="EndTime" separator=" " datatype="DateTime" format="yyyy-MM-dd HH:mm:ss">
            <field-locator locator="MQCAMO_END_DATE" locator-type="Label"/>
            <field-locator locator="MQCAMO_END_TIME" locator-type="Label"/>
        </field>
        <field name="CommandLevel" locator="MQIA_COMMAND_LEVEL" locator-type="Label"/>
        <field name="SequenceNumber" locator="MQIACF_SEQUENCE_NUMBER" locator-type="Label"/>
        <field name="ApplName" locator="MQCACF_APPL_NAME" locator-type="Label"/>
        <field name="ApplType" locator="MQIA_APPL_TYPE" locator-type="Label"/>
        <field name="ProcessId" locator="MQIACF_PROCESS_ID" locator-type="Label"/>
        <field name="UserId" locator="MQCACF_USER_IDENTIFIER" locator-type="Label"/>
        <field name="ApiCallerType" locator="MQIACF_API_CALLER_TYPE" locator-type="Label"/>
        <field name="ApiEnv" locator="MQIACF_API_ENVIRONMENT" locator-type="Label"/>
        <field name="ApplFunction" locator="MQCACF_APPL_FUNCTION" locator-type="Label"/>
        <field name="ApplFunctionType" locator="MQIACF_APPL_FUNCTION_TYPE" locator-type="Label"/>
        <field name="TraceDetail" locator="MQIACF_TRACE_DETAIL" locator-type="Label"/>
        <field name="TraceDataLength" locator="MQIACF_TRACE_DATA_LENGTH" locator-type="Label"/>
        <field name="PointerSize" locator="MQIACF_POINTER_SIZE" locator-type="Label"/>
        <field name="Platform" locator="MQIA_PLATFORM" locator-type="Label"/>
        <field name="Correlator" locator="MQBACF_CORREL_ID" locator-type="Label" datatype="Binary"/>
    </parser>

    <stream name="WmqStreamPCF" class="com.jkoolcloud.tnt4j.streams.inputs.WmqStreamPCF">
        <property name="Host" value="[YOUR_MQ_HOST]"/>
        <property name="Port" value="[YOUR_MQ_PORT]"/>
        <property name="QueueManager" value="[YOUR_QM_NAME]"/>
        <property name="Queue" value="[YOUR_Q_NAME]"/>
        <property name="StripHeaders" value="false"/>

        <parser-ref name="PCFEventsParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that WmqStreamPCF referencing PCFEventsParser shall be used. Stream takes MQ message from QM, transforms it to PCF message and passes it to parser.

Host property defines MQ server host name or IP. Port property defines MQ server port.

The QueueManager property defines the name of queue manager, and the Queue property defines the name of queue to get messages.

StripHeaders property states that MQ message headers shall be preserved.

PCFEventsParser is of type ActivityPCFParser meaning that it will parse PCF messages.

The TranslateNumValues property defines that parser should translate resolved numeric values to the corresponding MQ constant names, if possible.

WMQ Trace Events Streaming

This sample shows how to stream activity events received over IBM MQ as MQ Trace Events.

When configuring mqat.ini application specific stanza or setting the default values for subscriptions, the following values are advised:

  • TraceLevel should be set to MEDIUM or HIGH.
  • ActivityInterval, ActivityCount, SubscriptionDelivery and StopOnGetTraceMsg default values are sufficient in most cases.
  • TraceMessageData should be set to a value sufficient to capture required data (it can be 0 for no payload capture).

Sample files can be found in the samples/ibm-mq-trace-events directory (tnt4j-streams-wmq module).

Stream configuration states that WmqActivityTraceStream referencing TraceEventsParser shall be used. Stream takes MQ message from QM, transforms it to PCF message and passes it to parser. If PCF message contains multiple MQ Trace Events, then multiple activity events will be made from it.

Host property defines MQ server host name or IP. Port property defines MQ server port.

QueueManager property defines name of queue manager.

Queue property defines name of queue to get messages.

TopicString property defines specific of generic channel or application to create a subscription without using a queue.

The StripHeaders property states that MQ message headers must be preserved.

TraceOperations property defines set of traced operation names (using RegEx or wildcard). MQ Trace Events referencing operations not covered by this set will be filtered out from activities stream.

ExcludedRC property defines set of excluded MQ traces reason codes (delimited using | character). MQ Trace Events having reason codes defined by this set will be filtered out from activities stream. Set entries may be defined using both numeric and MQ constant name values.

SuppressBrowseGets property defines flag indicating whether to exclude WMQ BROWSE type GET operation traces from streaming.

TraceEventsParser is of type ActivityPCFParser meaning that it will parse PCF messages containing MQ Trace Events data contained within MQCFGR PCF structures.

The TranslateNumValues property defines that parser should translate resolved numeric values to the corresponding MQ constant names, if possible.

OpenOptions property defines additional open options to be set, include value="MQSO_WILDCARD_CHAR" when using a generic value for TopicString.

Angulartics (AngularJS Tracing)

This sample shows how to stream JavaScript events traces from Angulartics. TNT4J-Angulartics-plugin sends trace data over HTTP request http://localhost:9595. Thus, to process this we will need HttpStream running on port 9595.

Sample files can be found in the samples/angular-js-tracing directory.

How to setup sample environment see Setting Up the Angulartics Plugin for AngularJS.

Sample trace data is available in messages.json file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="JSONPayloadParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="StartTime" locator="$.timestamp" locator-type="Label" datatype="Timestamp" units="Milliseconds"/>
        <field name="ResourceName" locator="$.url" locator-type="Label"/>
        <field name="Correlator" locator="$.sid" locator-type="Label"/>
        <field name="Correlator" locator="$.rid" locator-type="Label"/>
        <field name="EventName" locator="$.eventName" locator-type="Label"/>
        <field name="EventType" value="EVENT"/>
        <field name="ElapsedTime" locator="$.pageLoad" locator-type="Label" datatype="Number" format="#####0"/>
        <field name="Browser" locator="$.browser" locator-type="Label"/>
        <field name="EventProperties" locator="$.properties" locator-type="Label"/>
    </parser>

    <parser name="AngularticsReqParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="Transport" locator="ActivityTransport" locator-type="Label"/>
        <field name="MsgBody" locator="ActivityData" locator-type="Label">
            <parser-ref name="JSONPayloadParser"/>
        </field>
    </parser>

    <stream name="AngularticsHttpStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="9595"/>
        <parser-ref name="AngularticsReqParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that HttpStream, referencing AngularticsReqParser should be used.

HttpStream starts an HTTP server on the port defined using the Port property. The HaltIfNoParser property indicates that the stream should skip unparsable entries. The stream puts received request payload data as a byte[] into the map using the key ActivityData.

AngularticsReqParser by default converts byte[] in entry ActivityData to JSON format string and uses the stacked parser named JSONPayloadParser to parse the format.

JSONPayloadParser transforms received JSON data string to Map and fills in activity event fields values from that map.

AJAX

This sample shows how to stream JavaScript events traces from AJAX. TNT4J-AJAX-interceptor sends trace data over HTTP request http://localhost:9595. Thus, to process this we will need HttpStream running on port 9595.

Sample files can be found in the samples/ajax directory.

How to setup sample environment see, Intercepting AJAX Calls Using TNT4J-Streams

Sample trace data is available in messages.json file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="JSONPayloadParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="StartTime" locator="$.startOfLoading" locator-type="Label" datatype="Timestamp"
               units="Nanoseconds"/>
        <field name="ResourceName" locator="$.url" locator-type="Label"/>
        <field name="EventName" locator="$.eventType" locator-type="Label"/>
        <field name="Message" locator="$.message" locator-type="Label"/>
        <field name="Tag" locator="$.eventType" locator-type="Label"/>

        <field name="CompCode" locator="$.statuss" locator-type="Label">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="$.statuss" locator-type="Label"/>

        <field name="EventType" value="EVENT"/>
        <field name="EndTime" locator="$.endOfLoading" locator-type="Label" datatype="Timestamp"
               units="Milliseconds"/>
        <field name="ElapsedTime" locator="$.elapsedTime" locator-type="Label" datatype="Number" format="#####0"/>
        <field name="ContentSize" locator="$.contentSize" locator-type="Label"/>
        <field name="IsError" locator="$.error" locator-type="Label"/>
        <field name="IsAborted" locator="$.abort" locator-type="Label"/>
    </parser>

    <parser name="AjaxEventParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="Transport" locator="ActivityTransport" locator-type="Label"/>
        <field name="MsgBody" locator="ActivityData" locator-type="Label">
            <parser-ref name="JSONPayloadParser"/>
        </field>
    </parser>

    <stream name="AjaxEventStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="9595"/>
        <parser-ref name="AjaxEventParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that HttpStream, referencing AjaxEventParser should be used.

HttpStream starts an HTTP server on the port defined using the Port property. The HaltIfNoParser property indicates that the stream should skip unparsable entries. The stream puts received request payload data as a byte[] into the map using the key ActivityData.

AjaxEventParser by default converts byte[] for entry ActivityData to JSON format string and uses stacked parser named JSONPayloadParser to parse it.

JSONPayloadParser transforms received JSON data string to Map and fills in activity event fields values from that map.

Node.js

This sample shows how to stream JavaScript events traces from Node.js. TNT4J-njsTrace-plugin sends trace data over HTTP request http://localhost:9595. Thus, to process this we will need HttpStream running on port 9595.

Sample files can be found in the samples/node.js directory.

How to setup sample environment, see Integrating njsTrace with TNT4J-Streams

Sample trace data is available in messages.json file.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="JSONPayloadParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="ElapsedTime" locator="$.span" locator-type="Label" datatype="Number" units="Milliseconds" required="false"/>
        <field name="ResourceName" separator=",">
            <field-locator locator="$.file" locator-type="Label"/>
            <field-locator locator="$.line" locator-type="Label"/>
        </field>

        <field name="Method" locator="$.name" locator-type="Label"/>
        <field name="Message" locator="$.returnValue" locator-type="Label" required="false"/>
        <field name="EventName" value="node.js Trace"/>
        <field name="EventType" locator="$.method" locator-type="Label"/>
        <field name="Correlator" locator="$.stack[*]" locator-type="Label" separator=","/>

        <field name="Exception" locator="$.exception" locator-type="Label"/>

        <field name="CompCode" locator="$.exception" locator-type="Label">
            <field-map source="false" target="SUCCESS"/>
            <field-map source="true" target="ERROR"/>
            <field-map source="Function timed out" target="ERROR"/>
        </field>

    </parser>

    <parser name="njstraceParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
        <field name="Transport" locator="ActivityTransport" locator-type="Label"/>
        <field name="MsgBody" locator="ActivityData" locator-type="Label">
            <parser-ref name="JSONPayloadParser"/>
        </field>
    </parser>

    <stream name="njstraceHttpStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
        <property name="HaltIfNoParser" value="false"/>
        <property name="Port" value="9596"/>
        <parser-ref name="njstraceParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that HttpStream, referencing njstraceParser should be used.

HttpStream starts an HTTP server on the port defined using the Port property. The HaltIfNoParser property indicates that the stream should skip unparsable entries. The stream puts received request payload data as a byte[] into the map using the key ActivityData.

njstraceParser by default converts byte[] for entry ActivityData to JSON format string and uses stacked parser named JSONPayloadParser to parse it.

JSONPayloadParser transforms received JSON data string to Map and fills in activity event fields values from that map.

Node.js Blocking Event Loop

This extended Node.js sample shows how to trace blocking event loop occurrences.

Sample files can be found in the samples/node.js-blocking-event-loop directory.

How to setup sample environment, see Using njsTrace with TNT4J-Streams for Node.js Performance Monitoring.

Sample stream configuration is same as in Node.js sample.

AWS CloudWatch Metrics over Kinesis FireHose

See AWS CloudWatch Metrics HTTP Endpoint and AWS CloudWatch Metric Streams with Kinesis Data Firehose for details.

Prometheus Remote-Write

See Prometheus Remote Write HTTP Endpoint  for details.

JAX-WS

This sample shows how to stream responses from JAX-WS (SOAP) services as SOAP compliant XML data.

Sample files can be found in the samples/ws-stream directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-ws.xsd">

    <parser name="WSResponseParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
        <property name="Namespace" value="s=http://schemas.xmlsoap.org/soap/envelope/"/>
        <property name="Namespace" value="a=http://schemas.datacontract.org/2004/07/"/>
        <property name="Namespace" value="i=http://www.w3.org/2001/XMLSchema-instance"/>
        <property name="Namespace" value="b=http://tempuri.org/"/>

        <field name="EventType" value="Event"/>
        <field name="ApplName" value="weather"/>
        <field name="GeoLocation" separator=",">
            <field-locator
                    locator="/s:Envelope/s:Body/b:GetCurrentWeatherInformationByStationIDResponse/b:GetCurrentWeatherInformationByStationIDResult/a:Latitude"
                    locator-type="Label"/>
            <field-locator
                    locator="/s:Envelope/s:Body/b:GetCurrentWeatherInformationByStationIDResponse/b:GetCurrentWeatherInformationByStationIDResult/a:Longitude"
                    locator-type="Label"/>
        </field>
        <field name="Temperature"
               locator="/s:Envelope/s:Body/b:GetCurrentWeatherInformationByStationIDResponse/b:GetCurrentWeatherInformationByStationIDResult/a:TemperatureInFahrenheit"
               locator-type="Label"/>
        <field name="Humidity"
               locator="/s:Envelope/s:Body/b:GetCurrentWeatherInformationByStationIDResponse/b:GetCurrentWeatherInformationByStationIDResult/a:RelativeHumidity"
               locator-type="Label"/>
        <field name="Wind Speed"
               locator="/s:Envelope/s:Body/b:GetCurrentWeatherInformationByStationIDResponse/b:GetCurrentWeatherInformationByStationIDResult/a:WindSpeedInMPH"
               locator-type="Label"/>
    </parser>

    <stream name="WSSampleStream" class="com.jkoolcloud.tnt4j.streams.inputs.WsStream">
        <property name="HaltIfNoParser" value="false"/>

        <scenario name="Sample WS stream scenario">
            <step name="Step 1"
                  url="http://wsdot.wa.gov/traffic/api/WeatherInformation/WeatherInformation.svc">
                <schedule-simple interval="35" units="Seconds" repeatCount="10"/>
                <request>
                    <![CDATA[
                        SOAPAction:http://tempuri.org/IWeatherInformation/GetCurrentWeatherInformationByStationID
                        <tem:GetCurrentWeatherInformationByStationID xmlns:tem="http://tempuri.org/">
                            <tem:AccessCode>aeb652b7-f6f5-49e6-9bdb-e2b737ebd507</tem:AccessCode>
                            <tem:StationID>1909</tem:StationID>
                        </tem:GetCurrentWeatherInformationByStationID>
                    ]]>
                </request>
            </step>
        </scenario>

        <parser-ref name="WSResponseParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that WsStream referencing WSResponseParser shall be used. Stream takes response received SOAP-compliant XML string and passes it to the parser.

HaltIfNoParser property indicates that the stream should skip unparseable entries.

The streaming scenario consists of one step defining service request URL, scheduler definition: 10 times at 35 seconds interval. request definition contains SOAP request header SOAPAction and SOAP request body data tem:GetCurrentWeatherInformationByStationID.

WSResponseParser maps XML data values to activity event fields GeoLocation, Temperature, Humidity and Wind Speed. Parser property Namespace adds additional namespaces required to parse received JAX-WS XML response using XPath.

JAX-RS

JAX-RS JSON

This sample shows how to stream responses from JAX-RS as JSON data.

Sample files can be found in the samples/restful-stream-json directory. See Weather Forecast Streaming Sample for information on usage and modifying the parser for your use.

Sample stream configuration:

To use this weather forecast sample, you will have to sign up for a free subscription to one of the weather services at OpenWeatherMap (for example, Current Weather Data or 5-day Forecast) and get an API key (APPID), which must be substituted (replace placeholder <YOUR_APP_TOKEN>) in the URL line of streams config. 

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-ws.xsd">

    <parser name="RESTResponseParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="EventType" value="Event"/>
        <field name="ApplName" value="weather"/>
        <field name="Location" locator="$.name" locator-type="Label"/>
        <field name="GeoLocation" separator=",">
            <field-locator locator="$.coord.lon" locator-type="Label"/>
            <field-locator locator="$.coord.lat" locator-type="Label"/>
        </field>
        <field name="Temperature" locator="$.main.temp" locator-type="Label"/>
        <field name="Humidity" locator="$.main.humidity" locator-type="Label"/>
        <field name="Wind Speed" locator="$.wind.speed" locator-type="Label"/>
    </parser>

    <stream name="RESTfulSampleJSONStream" class="com.jkoolcloud.tnt4j.streams.inputs.RestStream">
        <property name="HaltIfNoParser" value="false"/>

        <scenario name="Sample REST stream scenario">
            <step name="Step Kaunas"
                  url="http://api.openweathermap.org/data/2.5/weather?q=Kaunas&amp;APPID=<YOUR_APP_TOKEN>&amp;units=metric"
                  method="GET">
                <schedule-cron expression="0/15 * * * * ? *"/>
            </step>
            <step name="Step Vilnius"
                  url="http://api.openweathermap.org/data/2.5/weather?q=Vilnius&amp;APPID=<YOUR_APP_TOKEN>&amp;units=metric"
                  method="GET">
                <schedule-cron expression="0/30 * * * * ? *"/>
            </step>
            <step name="Step Klaipeda"
                  url="http://api.openweathermap.org/data/2.5/weather?q=Klaipeda&amp;APPID=<YOUR_APP_TOKEN>&amp;units=metric"
                  method="GET">
                <schedule-simple interval="45" units="Seconds" repeatCount="10"/>
            </step>
        </scenario>

        <parser-ref name="RESTResponseParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that RestStream referencing RESTResponseParser shall be used. Stream takes response received JSON string and passes it to parser.

HaltIfNoParser property indicates that the stream should skip unparseable entries.

Streaming scenario consists of three steps, defining service request URL's, request methods (all GET), and scheduler definitions.

  • Step named Step Kaunas defines a Cron scheduler expression stating: invoke every 15 seconds.
  • Step named Step Vilnius defines a  Cron scheduler expression stating: invoke every 30 seconds.
  • Step named Step Klaipeda defines a simple scheduler expression stating: 10 times at 45 seconds interval.

RESTResponseParser maps JSON data values to activity event fields Location, GeoLocation, Temperature, Humidity and Wind Speed. Parser property ActivityDelim indicates that whole parsed string represents single JSON data package.

JAX-RS XML

This sample shows how to stream responses from JAX-RS as XML data.

Sample files can be found in the samples/restful-stream-xml directory. See Weather Forecast Streaming Sample for information on usage and modifying the parser for your use.

Sample stream configuration:

To use this weather forecast sample, you will have to sign up for a free subscription to one of the weather services at OpenWeatherMap (for example, Current Weather Data or 5-day Forecast) and get an API key (APPID), which must be substituted (replace placeholder <YOUR_APP_TOKEN>) in the URL line of streams config. 

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-ws.xsd">

    <parser name="RESTResponseParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
        <field name="EventType" value="Event"/>
        <field name="ApplName" value="weather"/>
        <field name="Location" locator="/current/city/@name" locator-type="Label"/>
        <field name="GeoLocation" separator=",">
            <field-locator locator="/current/city/coord/@lon" locator-type="Label"/>
            <field-locator locator="/current/city/coord/@lat" locator-type="Label"/>
        </field>
        <field name="Temperature" locator="/current/temperature/@value" locator-type="Label"/>
        <field name="Humidity" locator="/current/humidity/@value" locator-type="Label"/>
        <field name="Wind Speed" locator="/current/wind/speed/@value" locator-type="Label"/>
    </parser>

    <stream name="RESTfulSampleXMLStream" class="com.jkoolcloud.tnt4j.streams.inputs.RestStream">
        <property name="HaltIfNoParser" value="false"/>

        <scenario name="Sample REST stream scenario">
            <step name="Step Kaunas"
                  url="http://api.openweathermap.org/data/2.5/weather?q=Kaunas&amp;APPID=<YOUR_APP_TOKEN>&amp;units=metric&amp;mode=xml"
                  method="GET">
                <schedule-cron expression="0/15 * * * * ? *"/>
            </step>
            <step name="Step Vilnius"
                  url="http://api.openweathermap.org/data/2.5/weather?q=Vilnius&amp;APPID=<YOUR_APP_TOKEN>&amp;units=metric&amp;mode=xml"
                  method="GET">
                <schedule-cron expression="0/30 * * * * ? *"/>
            </step>
            <step name="Step Klaipeda"
                  url="http://api.openweathermap.org/data/2.5/weather?q=Klaipeda&amp;APPID=<YOUR_APP_TOKEN>&amp;units=metric&amp;mode=xml"
                  method="GET">
                <schedule-simple interval="45" units="Seconds" repeatCount="10"/>
            </step>
        </scenario>

        <parser-ref name="RESTResponseParser"/>
    </stream>
</tnt-data-source>

Stream configuration states that RestStream referencing RESTResponseParser shall be used. Stream takes response received XML string and passes it to parser.

HaltIfNoParser property indicates that the stream should skip unparseable entries.

Streaming scenario consists of three steps defining service request URL's, request methods (all GET), and scheduler definitions.

  • Step named Step Kaunas defines a Cron scheduler expression stating: invoke every 15 seconds.
  • Step named Step Vilnius defines a Cron scheduler expression stating: invoke every 30 seconds.
  • Step named Step Klaipeda defines a simple scheduler expression stating: 10 times at 45 seconds interval.

RESTResponseParser maps XML data values to activity event fields Location, GeoLocation, Temperature, Humidity and Wind Speed.