Stream Elements Filtering: Using Cached Field Values

This article explains how to use caching in TNT4J-Streams to store and reuse field values from previously streamed activities, making it easier to handle related events.

If you haven’t already, check the article on using dynamic locators as another way to filter stream elements.

TNT4J-Streams provides temporary storage (e.g. cache) for a resolved activity field's values. It is useful when there are some related activities streamed, and particular jKool-prepared activity entity requires data values from previously streamed activities.

Sample streamed values caching configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="EventParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
        <property name="ActivityDelim" value="EOF"/>

        <field name="EventType" value="EVENT"/>

        <field name="EventName" locator="$.event" locator-type="Label"/>
        <field name="LastEvent" locator="CachedEventName" locator-type="Cache"/>

        <field name="Correlator" locator="$.transaction" locator-type="Label"/>
        <field name="Transaction" locator="$.transaction" locator-type="Label"/>

        <field name="SecretValue" locator="$.secret" locator-type="Label"/>
        <field name="LastSecret" locator="SecretCache" locator-type="Cache"/>

        <field name="Message" locator="$.message" locator-type="Label"/>
        <field name="Resource" locator="$.resource" locator-type="Label"/>
    </parser>

    <cache>
        <property name="MaxSize" value="300"/>
        <property name="ExpireDuration" value="15"/>

        <entry id="CachedEventName">
            <key>EventName</key>
            <value>${EventName} in ${Transaction}</value>
        </entry>
        <entry id="SecretCache">
            <key>${Transaction}Secret</key>
            <value>${SecretValue}</value>
        </entry>
    </cache>

    <stream name="MultipleEvents" class="com.jkoolcloud.tnt4j.streams.inputs.CharacterStream">
        <property name="FileName" value="./tnt4j-streams-core/samples/cached-values/event*.json"/>

        <parser-ref name="EventParser"/>
    </stream>

</tnt-data-source>

Sample configuration defines stream MultipleEvents reading data from JSON files using filename pattern event*.json and referencing parser EventParser. Stream defines two cache-related properties MaxSize and ExpireDuration. Definitions of those properties can be found in chapter Stream cache related parameters.

The stream definition has a cache section, defining stored cache entries. Entry key and value can be configured using static values (e.g., <key>EventName</key>), dynamic activity value references (e.g., <value>${SecretValue}</value>) referencing an activity entity field name, or combined static and dynamic values (e.g., <value>${EventName} in ${Transaction}</value>). Using dynamic value references in <key> definition will result in multiple cache entries filled in with values resolved from streamed activity data.

Parser EventParser has two fields with locator-type="Cache". This means that values from those fields are resolved from stream cache entries referenced by locator="SecretCache" where the locator value refers to a cache entry identifier. With reference to the cache entry, the stream uses the entry key definition and pre-fills it with current activity entity fields values generating the actual cache key and mapping it to a particular cached value.