Stream Data Source Configuration – meshIQ

Streams can be configured using an XML document with the root element tnt-data-source.
The XML schema definition is provided in tnt-data-source.xsd, located in the project’s config directory.

You can point to the streams data source configuration file in either of the following ways.

A) Program argument -f:

tnt4j-streams -f:./tnt4j-streams-core/samples/single-log/tnt-data-source.xml

B) JVM system property tnt4j.streams.config

-Dtnt4j.streams.config=./tnt4j-streams-core/samples/single-log/tnt-data-source.xml

Configuration Validation (Optional)

XSD validation: TNT4J-Streams can validate the tnt-data-source configuration XML against the XSD schema. Any validation failures are listed in the TNT4J-Streams log as WARNING entries. Enable XML–XSD validation with:

-Dtnt4j.streams.validate.config.xsd=true

Script expression validation (enabled by default): By default, TNT4J-Streams validates transformations, filters, and matcher script expressions defined in the configuration. Any failures are listed as ERROR entries. To disable script-expression validation use system property tnt4j.streams.validate.config.exp:

    -Dtnt4j.streams.validate.config.exp=false

Other Program Arguments

-p — Used in conjunction with PipedStream; only the parsers configuration from the <tnt-data-source/> definition is used. See OS piped stream.
-z: — Defines the configuration file for ZooKeeper–based TNT4J-Streams configuration. See Loading ZooKeeper stored configuration data and Streams registry.

sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="TokenParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityTokenParser">
        <property name="FieldDelim" value="|"/>
        <field name="StartTime" locator="1" format="dd MMM yyyy HH:mm:ss" locale="en-US"/>
        <field name="ServerIp" locator="2"/>
        <field name="ApplName" value="orders"/>
        <field name="Correlator" locator="3"/>
        <field name="UserName" locator="4"/>
        <field name="EventName" locator="5"/>
        <field name="EventType" locator="5">
            <field-map source="Order Placed" target="START"/>
            <field-map source="Order Received" target="RECEIVE"/>
            <field-map source="Order Processing" target="OPEN"/>
            <field-map source="Order Processed" target="SEND"/>
            <field-map source="Order Shipped" target="END"/>
        </field>
        <field name="MsgValue" locator="8"/>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="FileName" value="orders.log"/>
        <!--<property name="RangeToStream" value="1:"/>-->

        <parser-ref name="TokenParser"/>
    </stream>
</tnt-data-source>

As you can see in the sample, there are two major configuration elements: parser and stream.
Because the configuration is read using a SAX parser, referenced entities must be declared before they are used. Note that the stream references the parser:

<stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
    <.../>
    <parser-ref name="TokenParser"/>
</stream>

Therefore, the sequence of configuration elements is critical and cannot be swapped.

Data Source Properties

A data source has these predefined properties:

DefaultTimeZone — Defines the default time zone used by the data source configuration.
DefaultLocale — Defines the default locale used by the data source configuration.

These properties should be defined directly under the opening <tnt-data-source> element, for example:

<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <property name="DefaultTimeZone" value="UTC"/>
    <property name="DefaultLocale" value="en-US"/>

    <.../>
</tnt-data-source>

This minimizes the need to repeat timezone and locale attributes for every field locator.
However, you can override these defaults by specifying values on individual field-locator attributes.

Property Value Substitution

Anywhere in the configuration, the <property> element supports dynamic value resolution from these sources (in the following order; resolution stops on the first non-null value):

Java system properties
OS environment variables (property-name case is ignored)
TNT4J properties

Define a dynamically resolved property using this pattern:

<property name="PropName" value="${env.property.name}"/>

Where env.property.name is the property name from one of the sources above. Examples:

    <!-- Java system properties -->
<property name="UsedJava" value="${java.version}"/>
        <!-- OS environment variables -->
<property name="JavaHome" value="${JAVA_HOME}"/>
        <!-- TNT4J properties -->
<property name="SinkLogFile" value="${event.sink.factory.Filename}"/>

You can also use dynamic resolution for fragments of a value:

<property name="FileName" value="${user.dir}/data/access.log"/>