Parsers Configuration

This article covers how to configure parsers in meshIQ/TNT4J-Streams — detailing generic parser settings, available parser types (JSON, XML, JMS, etc.), and configuration options for parsing raw activity data.

Sample:

<parser name="TestParser" class="com.my.company.ActivityTestParser" manualFieldsOrder="true" default-data-type="Generic" default-emptyAsNull="false">
    <.../>
</parser>

• UseActivityDataAsMessageForUnset — Flag indicating whether Raw activity data shall be put into field Message if there is no mapping defined for that field in stream parser configuration or value was not resolved by parser from Raw activity data. NOTE: it is recommended to use it for DEBUGGING purposes only. For a production version of your software, remove this property form stream parser configuration. Default value — false. (Optional)

• ActivityDelim — Defines activities delimiter symbol/token used by parsers. Value can be one of: EOL — end of line, or EOF — end of file/stream. Default value — EOL. (Optional)

• DelimiterRule — Defines activities delimiter handling rules:

  • BEGINNING — delimiter marks beginning of raw activity data. Delimiter itself gets included into raw activity data. Aliases — BEGIN, STARTING, START FROM
  • TERMINATING — delimiter marks end of RAW activity data. Delimiter itself gets included into raw activity data. Aliases — TERMINATE, ENDING, END, STOP, TO
  • DELIMITING  — delimiter is ordinary data chunks separator. Delimiter itself gets omitted from raw activity data. Aliases — SPLIT, DELIM, DELIMIT, SEPARATE, DIVIDE, DIV
  • Default value — TERMINATE. (Optional)
• RequireDefault — Indicates that all parser fields/locators by default are required to resolve to non-null values. Default value — false. (Optional) The field attribute required="true" (or false) may be used to take precedence over the RequireDefault property. See required attribute definition in TNT4J Events field mappings.

• AutoArrangeFields — Flag indicating parser fields shall be automatically ordered by parser to ensure references sequence. When false, fields shall maintain user parser configuration-defined order. NOTE: it is alias for parser configuration attribute manualFieldsOrder. Default value — true. (Optional)

• Set of user defined parser context properties. To define parser context property add ctx: prefix to property name. These properties are not used directly by parser itself, but can be used in parser configuration over dynamic locators or variable expressions to enrich parsing context. (Optional)

Sample:

<property name="UseActivityDataAsMessageForUnset" value="true"/>
<property name="ActivityDelim" value="EOF"/>
<property name="RequireDefault" value="true"/>
<property name="AutoArrangeFields" value="false"/>

• $DATA$ — Allows setting complete activity data as field value and redirect it to stacked parser if such is defined for that field.

• $METADATA$ — Allows accessing activity bound meta-data map. Activity meta-data binding is stream specific: some streams may not provide any meta-data. To get list of stream provided meta-data entries, see particular stream documentation (both JavaDoc and readme).

• $RAWDATA$ — Allows setting complete RAW activity data (one before pre-parsing other preparations) as field value and redirect it to stacked parser if such is defined for that field.

• Field wildcard locators having * symbol within field name (e.g., FieldNameFragment*) are used to capture map of activity entity field values, where map entry key is field name and value is field value.

• FieldDelim — Fields separator. Default value — ,. (Optional)

• ValueDelim — Value delimiter. Default value — =. (Optional)

• Pattern — Pattern used to determine which types of activity data string this parser supports. When null, all strings are assumed to match the format supported by this parser. Default value — null. (Optional)

• StripQuotes — Whether surrounding double quotes should be stripped off. Default value — true. (Optional)

• EntryPattern — Pattern used to split data into name/value pairs. It should define two RegEx groups named key and value used to map data contained values to name/value pair. If any other named groups are defined, then group name is used as the key for the map entry. NOTE: this parameter takes preference on FieldDelim and ValueDelim parameters. (Optional)

Sample:

<property name="FieldDelim" value=";"/>
<property name="ValueDelim" value="-"/>
<property name="Pattern" value="(\S+)"/>
<property name="StripQuotes" value="false"/>
<property name="EntryPattern"><![CDATA[:(?<key>.*?):(?<value>.[^:]+)]]></property>

Also see Activity map parser regarding higher level parser configuration.

• Pattern — Contains the regular expression pattern that each data item is assumed to match. (Required)

• MatchStrategy — Defines pattern created matcher comparison strategy used against input data string. Value can be one of:
  MATCH — pattern should match complete input string, or
  FIND — pattern has to match sub-sequence within input string.
  Default value — MATCH. (Optional)

Sample:

• Index-capturing groups:

      <property name="Pattern" value="((\S+) (\S+) (\S+))"/>
      <property name="MatchStrategy" value="FIND"/>

• Named-capturing groups:

      <property name="Pattern"><![CDATA[
          (?<CoID>.*)\.(?<ProcessArea>.*)\.(?<InterfaceID>.*)\.(?<HopNr>.*)
      ]]></property>
      <property name="MatchStrategy" value="MATCH"/>

  When MatchStrategy=FIND is used and regex returns multiple matches, it is possible to access an individual match group by defining the Label type locator match index ( 1 can be omitted since it is the default) and a group descriptor (an index or name), following the match index after a period delimiter ., e.g.,

  • locator="2.2" locator-type="Label" will return the group indexed by 2 for match 2.

  • locator="3.groupName" locator-type="Label" will return the group named groupName for match 3.

• Expression type locators:

      <field name="EventName">
          <field-locator locator-type="Expression"><![CDATA[<table id="(.[^"]*)"]]></field-locator>
      </field>

  Resolves individual field value within whole activities data string.

• Wildcard locators:

    <property name="Pattern"><![CDATA[<tbody[^>]*?[^>]*?>((?s).*?)</tbody>]]></property>
    <.../>
    <field name="TableBodyData" locator="*.0" locator-type="Label" transparent="true" split="true" datatype="AsInput">
        <parser-ref name="TableContentParser" aggregation="Relate"/>
    </field>

  This sample crops all <tbody> content from HTML document. Both match and group (index or name) tokens can be wildcard *, e.g.:

  • *.* — returns all groups for all matches

  • *.1 — returns group indexed 1 for all matches

  • 1.* — returns match indexed 1 all groups

  • 2.*Number — returns match indexed 2 all groups name ending Number

Also see Generic parser parameters.

• FieldDelim — Fields separator. Default value — ,. (Optional)

• Pattern — Pattern used to determine which types of activity-data strings this parser supports. When null, all strings are assumed to match the format supported by this parser. Default value — null. (Optional)

• StripQuotes — Whether surrounding double quotes should be stripped off. Default value — true. (Optional)

Sample:

<property name="FieldDelim" value=";"/>
<property name="Pattern" value="(\S+)"/>
<property name="StripQuotes" value="false"/>

Also see Generic parser parameters.

This parser uses XPath expressions as field locators. You may also use TNT4J-Streams predefined custom XPath functions.

• Namespace — Additional XML namespace mappings. Default value — null. (Optional)

• NamespaceAware — Indicates that the parser has to provide support for XML namespaces. Default value — true. (Optional)

• IsolateNodeXML — Indicates that the parser shall isolate the provided DOM Node XML and use it as a standalone document, omitting references to the parent node/document. Default value — true. (Optional)

• FallbackToParent — Indicates that the parser shall try to resolve a value using the parent document scope if it was unable to resolve the value using the relative context (when the XPath expression has a “wider” scope than the “current” node provides). Default value — true. (Optional)

Sample:

<property name="Namespace" value="xsi=http://www.w3.org/2001/XMLSchema-instance"/>
<property name="Namespace" value="tnt4j=https://xray.meshiq.com/xray/xsds"/>
<property name="NamespaceAware" value="false"/>

XML parser provides the following activity metadata entries (accessible using the $METADATA$ locator):

• MD_XML_PARENT_NODE_NAME — Parent node name when the parser has IsolateNodeXML set to true.

Also see Generic parser parameters.

Also see Activity XML parser and Generic parser parameters.

For MQ messages it is possible to calculate a message signature from message fields. To initiate signature calculation as a field value, the field tag value-type attribute value has to be set to signature.

Sample of field definition for signature calculation: 

<field name="TrackingId" separator="#!#" value-type="signature">
    <field-locator locator="/messaging_operation/MsgType" locator-type="Label" datatype="Number"/>
    <field-locator locator="/messaging_operation/MsgFormat" locator-type="Label"/>
    <field-locator locator="/messaging_operation/MsgId" locator-type="Label" datatype="Binary" format="hexBinary"/>
    <field-locator locator="/messaging_operation/MsgUser" locator-type="Label">
        <field-transform name="UserIdLowerCase" lang="groovy">
            StringUtils.lowerCase($fieldValue)
        </field-transform>
    </field-locator>
    <field-locator locator="/messaging_operation/MsgPutApplType" locator-type="Label"/>
    <field-locator locator="/messaging_operation/MsgPutApplName" locator-type="Label"/>
    <field-locator locator="/messaging_operation/MsgPutDate" locator-type="Label"/>
    <field-locator locator="/messaging_operation/MsgPutTime" locator-type="Label"/>
    <field-locator locator="/messaging_operation/Correlator" locator-type="Label"/>
</field>

• LogPattern — Access-log pattern. (Optional, if RegEx Pattern property is defined)

• ConfRegexMapping — Custom log-pattern token (must be wildcard or RegEx-compliant string) and RegEx mapping. (Optional, applicable only if LogPattern is used)

Sample: Using a log pattern with custom mappings:

<property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>
<property name="ConfRegexMapping" value="%h=(\S+)"/>
<property name="ConfRegexMapping" value="%*s=(\d{3})"/>
<property name="ConfRegexMapping" value="%*r=(((\S+) (.*?)( (\S+)|()))|(-))"/>
<property name="ConfRegexMapping" value="%*i=(.*?)"/>

Or defining a complete RegEx pattern to match log lines:

<property name="Pattern"
          value="^(\S+) (\S+) (\S+) \[([\w:/]+\s[+\-]\d{4})\] &quot;(((\S+) (.*?)( (\S+)|()))|(-))&quot; (\d{3}) (\d+|-)( (\S+)|$)"/>

<property name="ConfRegexMapping"><![CDATA[%*r=(?<request>((?<method>\S+) (?<uri>.*?)( (?<version>\S+))?)|(-))]]></property>

Also see Generic parser parameters.

• LocPathDelim — locator path delimiter in a map. An empty value means the locator value should not be delimited into path elements. Default value — .. (Optional)

Sample:

<property name="LocPathDelim" value="/"/>

Also see Generic parser parameters.

Using locator path token value * (e.g., locator="*") you can make the parser take all map entries from that level and output them as activity entity fields/properties by using map entry data as follows:

• Map entry key — field/property name

• Map entry value — field/property value

Locator path token * can be omitted if the last path token resolves to java.util.Map type value. However, to get the complete map for the root path level you must define it locator="*" anyway, since the locator value can't be empty.

Using locator path token value # (e.g., locator="#") you can make the parser get all yet parser un-touched map entries from that level and output them as activity entity fields/properties by using map entry data as follows:

• Map entry key — field/property name

• Map entry value — field/property value

This allows the user to map part of the entries manually and the remainder automatically. Consider a map having the following entries:

entry1: key=key1, value=value1
entry2: key=key2, value=value2
entry3: key=key3, value=value3

Then, using parser configuration:

<field name="Message" locator="key2" locator-type="Label"/>
<field name="AllRestMapEntries" locator="#" locator-type="Label"/>

Message=value2
key1=value1
key3=value3

• .size() (or .length()) — returns size of prefix locator expression resolved value being of type: array, collection or map. Example: events.size() will return size of events list.

• .keys() — returns key set for prefix locator expression resolved map. Example: timeseries.keys() will return keys of timeseries map.

• .values() — returns values collection for prefix locator expression resolved map. Example: timeseries.values() will return values collection of timeseries map.

• .entries() — returns entry set for prefix locator expression resolved map. Example: timeseries.entries() will return entry set of timeseries map.

We have two classes implementing JSON parser. Difference is type of expressions used for parsing:

• com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser - uses JSONPath expressions for parsing 
  Sample file: <tnt4j-streams>/tnt4j-streams-core/src/main/java/com/jkoolcloud/tnt4j/streams/parsers/ActivityJsonParser.java
• com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonJMESPathParser - uses JMESPath expressions for parsing
  Sample file: <tnt4j-streams>/tnt4j-streams-core/src/main/java/com/jkoolcloud/tnt4j/streams/parsers/ActivityJsonJMESPathParser.java

Configuration properties:

• ReadLines — indicates that complete JSON data package is single line. Default value — true. (Optional, deprecated — use ActivityDelim of Generic parser instead)

• List of DeserializationFeature.[FEATURE_NAME] — defines set of Jackson Object Mapper's deserialization configuration features. See DeserializationFeature. (Optional)

• List of MapperFeature.[FEATURE_NAME] — defines set of Jackson Object Mapper's mapping configuration features. See MapperFeature. (Optional)

• List of JsonParser.[FEATURE_NAME] — defines set of Jackson Object Mapper's parser configuration features. See JsonParser.Feature. (Optional)

• List of Option.[OPTION_NAME] — defines set of JsonPath configuration options. See Option. (Optional)

Sample:

<property name="ReadLines" value="false"/>
<!-- Jackson Object Mapper's Deserialization features -->
<property name="DeserializationFeature.USE_BIG_DECIMAL_FOR_FLOATS" value="false"/>
<property name="DeserializationFeature.USE_BIG_INTEGER_FOR_INTS" value="false"/>
<property name="DeserializationFeature.USE_LONG_FOR_INTS" value="false"/>
<property name="DeserializationFeature.USE_JAVA_ARRAY_FOR_JSON_ARRAY" value="false"/>
<property name="DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES" value="true"/>
<.../>
<!-- Jackson Object Mapper's mapping features -->
<property name="MapperFeature.USE_ANNOTATIONS" value="true"/>
<property name="MapperFeature.USE_GETTERS_AS_SETTERS" value="true"/>
<property name="MapperFeature.PROPAGATE_TRANSIENT_MARKER" value="false"/>
<property name="MapperFeature.AUTO_DETECT_CREATORS" value="true"/>
<property name="MapperFeature.AUTO_DETECT_FIELDS" value="true"/>
<.../>
<!-- Jackson Object Mapper's parser features -->
<property name="JsonParser.AUTO_CLOSE_SOURCE" value="true"/>
<property name="JsonParser.ALLOW_COMMENTS" value="false"/>
<property name="JsonParser.ALLOW_YAML_COMMENTS" value="false"/>
<property name="JsonParser.ALLOW_UNQUOTED_FIELD_NAMES" value="false"/>
<.../>
<!-- JsonPath configuration options -->
<property name="Option.DEFAULT_PATH_LEAF_TO_NULL"/>
<property name="Option.ALWAYS_RETURN_LIST"/>
<property name="Option.AS_PATH_LIST"/>
<.../>

Also see Generic parser parameters.

• ConvertToString — flag indicating whether to convert message payload byte[] data to string. Applicable to BytesMessage and StreamMessage. Default value — false. (Optional)

Sample:

<property name="ConvertToString" value="true"/>

Since JMS message parser is extension of Activity map parser, it supports map entry Wildcard locators:

• * — maps all JMS message resolved map entries to activity entity data, e.g.:

      <field name="AllMsgCustomProps" locator="MsgMetadata.CustomMsgProps.*" locator-type="Label"/>

  This will add all MsgMetadata.CustomMsgProps map entries as JKool activity entity fields/properties without any additional manual mapping, taking field/property name from map entry name and value from map entry value. The same way you can map all MsgMetadata map entries into activity entity fields/properties (note omitted locator .* token, see Wildcard locators for details):

      <field name="MsgMetadata" locator="MsgMetadata" locator-type="Label"/>

• # — maps set of unmapped JMS message resolved map entries to activity entity data, e.g.:

      <field name="Correlator" locator="MsgMetadata.Correlator" locator-type="Label"/>
      <field name="Destination" locator="MsgMetadata.Destination" locator-type="Label"/>
      <field name="MessageId" locator="MsgMetadata.MessageId" locator-type="Label"/>
      <field name="Priority" locator="MsgMetadata.Priority" locator-type="Label"/>
      <!-- automatically puts all unmapped message metadata map entries as custom activity properties -->
      <field name="AllRestMsgMetadataProps" locator="MsgMetadata.#" locator-type="Label"/>

  This will add all manually unmapped MsgMetadata map entries as JKool activity entity fields/properties, taking field/property name from map entry name and value from map entry value. 

Also see Activity map parser and Generic parser parameters.

See Generic parser parameters for general parser configuration. This parser does not have any additional configuration properties.

List of predefined locators for this parser:

• topic — topic this record is received from

• partition — partition from which this record is received

• offset — position of this record in the corresponding Kafka partition

• timestamp — timestamp of this record

• timestampType — timestamp type of this record

• serializedKeySize — size of the serialized, uncompressed key in bytes

• serializedValueSize — size of the serialized, uncompressed value in bytes

• key — record key

• value — record data

• headers — record headers iterable

• leaderEpoch — record leader epoch

See Generic parser parameters for general parser configuration. This parser does not have any additional configuration properties.

List of predefined locators for this parser:

• topic — topic name record is being sent to

• partition — partition identifier to which the record will be sent

• timestamp — record timestamp value

• key — record key

• value — record data

• headers — record headers iterable

• TranslateNumValues — indicates that parser should translate resolved numeric values to corresponding MQ constant names if possible and field/locator data type is String (meaning translated value can be assigned to field). If value of particular field should be left as number (e.g., ReasonCode), use field/locator attribute datatype="Number". Default value — true. (Optional)

Sample:

<property name="TranslateNumValues" value="false"/> 

<field-locator locator="MQGACF_ACTIVITY_TRACE.MQBACF_MSG_ID" locator-type="Label" datatype="String" charset="IBM500"/>

<field name="Message" locator="MQGACF_ACTIVITY_TRACE.MQBACF_MSG_ID" locator-type="Label" datatype="Binary">
    <field-transform name="MsgIdToString" lang="groovy"><![CDATA[
        Utils.getString($fieldValue, "IBM500")
    ]]>
    </field-transform>
</field>

<field name="MQTrace_CodedCharsetId" locator="MQGACF_ACTIVITY_TRACE.MQIA_CODED_CHAR_SET_ID" locator-type="Label" datatype="Number"/>
<field name="Message" locator="MQGACF_ACTIVITY_TRACE.MQBACF_MESSAGE_DATA" locator-type="Label" datatype="Binary">
<field-transform name="MsgPayloadToString" lang="groovy"><![CDATA[
    WmqUtils.getString($fieldValue, ${MQTrace_CodedCharsetId})
]]>
</field-transform>
</field>

<field name="MQTrace_CodedCharsetId" locator="MQGACF_ACTIVITY_TRACE.MQIA_CODED_CHAR_SET_ID" locator-type="Label" datatype="Number"/>
<field name="Message" locator="MQGACF_ACTIVITY_TRACE.MQBACF_MESSAGE_DATA" locator-type="Label" datatype="Binary">
<field-transform name="MsgPayloadToString" lang="groovy"><![CDATA[
    WmqUtils.getString($fieldValue, ${MQTrace_CodedCharsetId}, WmqUtils.MQ_BIN_STR_STRIP_DLH_XQH)
]]>
</field-transform>
</field>

A CCSID used in an expression should be a decimal number without a leading zero 0, which in Java means an octal representation of the number; most likely not what you wanted. Example: Test if a received MQ message is EBCDIC code page 37. If 037 (CCSID name alias) value is used, the comparison would be against decimal value 31, not 37, and would never match. Sample transformation:

<field-transform name="MsgEbcdicOrNot" lang="groovy"><![CDATA[
    ${MQTrace_CodedCharsetId} == 37 ? $fieldValue : "MSG NOT EBCDIC"
]]>
</field-transform>

Also see Generic parser parameters.

For MQ messages it is possible to calculate message signature from message fields. To initiate signature calculation as a field value, field tag value-type attribute value has to be set to signature.

Sample of field definition for signature calculation:

<field name="Correlator" value-type="signature">
    <field-locator locator="MQGACF_ACTIVITY_TRACE.MQIACF_MSG_TYPE" locator-type="Label" datatype="Number"/>
    <field-locator locator="MQGACF_ACTIVITY_TRACE.MQCACH_FORMAT_NAME" locator-type="Label"/>
    <field-locator locator="MQGACF_ACTIVITY_TRACE.MQBACF_MSG_ID" locator-type="Label" datatype="String" format="bytes"/>
    <field-locator locator="MQCACF_USER_IDENTIFIER" locator-type="Label">
        <field-transform name="UserIdLowerCase" lang="groovy">
            StringUtils.lowerCase($fieldValue)
        </field-transform>
    </field-locator>
    <field-locator locator="MQIA_APPL_TYPE" locator-type="Label"/>
    <field-locator locator="MQCACF_APPL_NAME" locator-type="Label"/>
    <field-locator locator="MQGACF_ACTIVITY_TRACE.MQCACF_PUT_DATE" locator-type="Label"/>
    <field-locator locator="MQGACF_ACTIVITY_TRACE.MQCACF_PUT_TIME" locator-type="Label"/>
    <field-locator locator="MQGACF_ACTIVITY_TRACE.MQBACF_CORREL_ID" locator-type="Label" datatype="Binary" required="false"/>
</field>

PCF message can have grouped parameters — all messages will have header MQCFH and may have MQCFGR type parameters.

• To access PCF message header fields use MQCFH expression with header field name delimited using . (e.g., MQCFH.CompCode).

• To access PCF message parameters use MQ constant name/value (e.g., MQCACF_APPL_NAME or 3024).

• To access inner MQCFGR (or inner-inner and so on) parameters use group parameter MQ constant name/value with grouped parameter MQ constant name/value delimited using . (e.g., MQGACF_ACTIVITY_TRACE.MQIACF_COMP_CODE).

• In case PCF parameter refers MQI structure, inner (or inner-inner and so on) structure fields can be accessed by adding . delimited MQI structure fields names to locator path after root MQI structure parameter identifier (e.g., MQGACF_ACTIVITY_TRACE.MQBACF_MQMD_STRUCT.MsgId or MQGACF_ACTIVITY_TRACE.MQBACF_MQCNO_STRUCT.clientConn.userIdentifier).

• Additionally PCFMessage may contain MQMD data copied from transport MQMessage. To access MQMD values use MQMD expression with field name (or relative PCF parameter constant) delimited using . (e.g., MQMD.PutApplName, MQMD.MQCACF_APPL_NAME or MQMD.3024).

This parser has no additional configuration properties.

Also see Activity map parser regarding higher level parser configuration.

This parser resolved data map may contain such entries:

• Date — resolved log entry date string

• Time — resolved log entry time string

• Process — resolved log entry process identifier

• User — resolved log entry user name

• Program — resolved log entry program (application) name

• Host — resolved log entry host name IBM MQ is running on

• Installation — resolved log entry IBM MQ installation name

• VRMF — resolved log entry running IBM MQ version descriptor

• QMgr — resolved log entry Queue manager error occurred on

• ErrCode — resolved log entry IBM MQ error code string

• ErrText — resolved log entry IBM MQ error message text

• Explanation — resolved log entry IBM MQ error explanation message text

• Action — resolved log entry IBM MQ error fix action message text

• Where — resolved log entry error descriptor location string containing source code file name and line number

IBM MQ versions starting 9.1 adds these additional entries:

• Severity — resolved log entry severity:
  'I' — INFO
  'W' — WARNING
  'E' — ERROR
  'S' — CRITICAL
  'T' — HALT

• TimeUTC — resolved log entry UTC time

• CommentInsert(X) — resolved log entry text attribute values, have been insert into log entry message, where (X) is attribute index

• ArithInsert(X) — resolved log entry numeric attribute values, have been insert into log entry message, where (X) is attribute index

• SupportedClass — defines class name of parser supported objects. Parser can have multiple definitions of this property. It is useful when just some specific set of objects has to be handled by this parser instead of all passed objects. (Optional)

Sample:

<property name="SupportedClass" value="org.apache.kafka.common.TopicPartition"/>
<property name="SupportedClass" value="com.jkoolcloud.tnt4j.streams.custom.interceptors.kafka.reporters.trace.KafkaTraceEventData"/>

Also see Generic parser parameters regarding higher level parser configuration.

This parser has no additional configuration properties.

Also see Generic parser parameters regarding higher level parser configuration.

This parser has no additional configuration properties.

Also see Activity map parser regarding higher level parser configuration.

This parser resolved data map may contain such entries:

• rfh2Folders — RFH2 folders data XML string. Root element for this XML is <rfh2Folders>. Further XPath based parsing can be processed by Activity XML parser

• jmsMsgPayload — JMS message payload data: de-serialized object or bytes if serialisation can't be done.

• UseFormattedCellValue — indicator flag stating to use formatted cell value (always String) as field/locator RAW data. When this flag is set to true — original cell value provided by Apache POI API is used e.g., making all numeric cells values as decimals (double) what is not very comfortable when entered cell value is integer. Default value — false. (Optional)

Sample:

<property name="UseFormattedCellValue" value="true"/>

Also see Generic parser parameters regarding higher level parser configuration.

• SQLJavaMapping — defines mapping from SQL type name (as String) to class (as Class.forName(String)) names in the Java programming language, e.g. NUMBER=java.lang.String. Parser can have multiple definitions og this property. It is useful when default JDBC driver mapping produces inaccurate result. IMPORTANT: if JDBC driver does not support java.sql.ResultSet.getObject(int, Map) or java.sql.ResultSet.getObject(String, Map) implementation, leave SQL-Java types map empty! (Optional)

Also see Generic parser parameters regarding higher level parser configuration.

The TNT4J-Streams architecture has a pre-parser entity that defines a data transformation algorithm to convert Raw activity data to a format supported by the stream’s parser. Pre-parsers must implement the ActivityDataPreParser interface.

It is possible to define multiple pre-parsers for the same parser instance. In that case, pre-parsers are applied sequentially (the sequence is defined by the order of the reference tags within the parser tag), where the input of the applied pre-parser is the output of the previous pre-parser.

What it does

• Retrieves XML data from a mixture of binary and string data (like RFH2).

• Makes an incomplete XML string fragment into a complete (DOM-valid) XML string.

Sample stream configuration using BinaryToXmlPreParser pre-parser

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <java-object name="RFH2ToXMLPreParser" class="com.jkoolcloud.tnt4j.streams.preparsers.BinaryToXmlPreParser">
        <!-- IF Raw ACTIVITY DATA STRING IS ENCODED - USE "format" PARAMETER. SEE BELOW -->
        <!--<param name="format" value="base64Binary" type="java.lang.String"/>-->
        <!-- IF RAW ACTIVITY DATA IS ENCODED USING CHARSET - USE "charsetName" PARAMETER. SEE BELOW -->
        <!--<param name="charsetName" value="UTF-16" type="java.lang.String"/>-->
    </java-object>

    <parser name="RFH2XMLParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
        <reference name="RFH2ToXMLPreParser"/>

        <field name="EventType" value="Event"/>
        <field name="ApplName" value="Sample"/>
        <field name="ADPCount" locator="/root/usr/ADPSegCont" locator-type="Label"/>
    </parser>

    <stream name="FileStream" class="com.jkoolcloud.tnt4j.streams.inputs.BytesInputStream">
        <property name="FileName" value="./tnt4j-streams-core/samples/XML-from-bin-data/RFH2.dump"/>
        <property name="RestoreState" value="false"/>

        <parser-ref name="RFH2XMLParser"/>
    </stream>
</tnt-data-source>

The sample configuration defines FileStream reading binary data from RFH2.dump. The stream refers to the activity XML parser RFH2XMLParser.

The RFH2XMLParser parser has a reference to RFH2ToXMLPreParser. This pre-parser builds valid XML from provided binary data having fragmented XML data like the RFH2 structure from IBM MQ.

If Raw activity data is not byte[] but appears to be encoded (e.g., BASE64), use the pre-parser property format to define the Raw activity data format. It may be one of the ActivityFieldFormatType enumeration values:

• base64Binary — BASE64 encoded binary data

• hexBinary — binary data represented as bytes HEX codes sequence

• string — binary data represented as plain string

• bytes — binary data as byte[] (default)

What it does

• Applies a user-defined transformation bean or script code to convert/alter parser input Raw activity data.

Used configuration properties

• id — transformation identifier. (Optional)

• lang — transformation script language. (Optional)

• script — code of the transformation script (can’t be mixed with beanRef). (Required)

• beanRef — transformation bean reference (can’t be mixed with script). (Required)

• useParserInput — flag indicating whether to use the parser-provided input read function. Default value — true. (Optional)

Sample stream configuration using TransformationPreParser pre-parser:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source-wmq_pcf.xsd">

    <java-object name="UnescapePreParser" class="com.jkoolcloud.tnt4j.streams.preparsers.TransformationPreParser">
        <property name="id" value="unescape"/>
        <property name="lang" value="groovy"/>
        <property name="useParserInput" value="false"/>
        <property name="script"><![CDATA[
            StringEscapeUtils.unescapeJava($fieldValue)
        ]]></property>
    </java-object>

    <parser name="XML_Data_Parser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityXmlParser">
        <reference name="UnescapePreParser"/>

        <.../>
    </parser>
</tnt-data-source>

Also see Field value transformations regarding the use of streamed data transformations.