Streaming Runtime Logs and HDFS Files in TNT4J-Streams – meshIQ

This article outlines sample configurations to capture log data in real time using TNT4J-Streams, with examples for log file polling and HDFS streaming.

Log File Polling

This sample shows how to stream Apache access log records as activity events from file which is used for logging at runtime. File polling technique may be used for any text file. File rolling is also supported.

Sample files can be found in samples/log-file-polling directory.

Sample stream configuration:

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">

    <parser name="AccessLogParserCommon" class="com.jkoolcloud.tnt4j.streams.custom.parsers.ApacheAccessLogParser">
        <property name="LogPattern" value="%h %l %u %t &quot;%r&quot; %s %b %D"/>

        <field name="Location" locator="1" locator-type="Index"/>
        <field name="UserName" locator="3" locator-type="Index"/>
        <field name="StartTime" locator="4" locator-type="Index" format="dd/MMM/yyyy:HH:mm:ss Z" datatype="DateTime" locale="en-US"/>
        <field name="EventType" value="SEND"/>
        <field name="EventName" locator="7" locator-type="Index"/>
        <field name="ResourceName" locator="8" locator-type="Index"/>
        <field name="CompCode" locator="12" locator-type="Index">
            <field-map source="100:206" target="SUCCESS" type="Range"/>
            <field-map source="300:308" target="WARNING" type="Range"/>
            <field-map source="400:417" target="ERROR" type="Range"/>
            <field-map source="500:511" target="ERROR" type="Range"/>
        </field>
        <field name="ReasonCode" locator="12" locator-type="Index"/>
        <field name="MsgValue" locator="13" locator-type="Index"/>
        <field name="ElapsedTime" locator="14" locator-type="Index" datatype="Number" format="#####0.000" locale="en-US" units="Seconds"/>

    </parser>

    <stream name="SampleFilePollingStream" class="com.jkoolcloud.tnt4j.streams.inputs.FileLineStream">
        <property name="FileName"
                  value="[PATH_TO_LOGS_REPOSITORY]/logs/localhost_access_log.*.txt"/>
        <property name="FilePolling" value="true"/>
        <property name="FileReadDelay" value="20"/>
        <property name="StartFromLatest" value="true"/>
        <parser-ref name="AccessLogParserCommon"/>
    </stream>
</tnt-data-source>

Stream configuration states that FilePollingStream referencing AccessLogParserCommon shall be used.

FileStream reads data from the access.log file. HaltIfNoParser property states that the stream should skip unparseable entries and not stop if such a situation occurs.

AccessLogParserCommon is the same as in the 'Apache Access log single file' sample, so refer to it for more details.

FileName property defines that the stream should watch for files matching localhost_access_log.*.txt wildcard pattern. This is needed to properly handle file rolling.

FilePolling property indicates that the polls files for changes.

FileReadDelay property indicates that file changes are streamed every 20 seconds.

StartFromLatest property indicates that the stream should start from the latest entry record in the log file. Setting this property to false would stream all log entries starting from the oldest file matching wildcard pattern.

Stream stops only when critical runtime error/exception occurs or application gets terminated.

HDFS

These samples show how to read or poll HDFS file contents. Samples are very similar to 'Log file polling' or the Apache Access Log Single File. The difference is that specialized stream classes are used.

Simple HDFS file streaming

Sample files can be found in tnt4j-streams/tnt4j-streams-hdfs/samples/hdfs-file-stream directory.

<stream name="SampleHdfsFileLineStream" class="com.jkoolcloud.tnt4j.streams.inputs.HdfsFileLineStream">
    <property name="FileName" value="hdfs://127.0.0.1:19000/log.txt*"/>
    <.../>
</stream>

To stream HDFS file lines HdfsFileLineStream shall be used. FileName is defined using URI starting hdfs://.

HDFS file polling

Sample files can be found in tnt4j-streams/tnt4j-streams-hdfs/samples/hdfs-log-file-polling directory.

<stream name="SampleHdfsFilePollingStream" class="com.jkoolcloud.tnt4j.streams.inputs.HdfsFileLineStream">
    <property name="FileName"
              value="hdfs://[host]:[port]/[path]/logs/localhost_access_log.*.txt"/>
    <property name="FilePolling" value="true"/>
    <.../>
</stream>

To poll HDFS file HdfsFileLineStream shall be used with property FilePolling value set to true. FileName is defined using URI starting hdfs://.

Stream stops only when critical runtime error/exception occurs or application gets terminated.

Zipped HDFS file streaming

Sample files can be found in tnt4j-streams/tnt4j-streams-hdfs/samples/hdfs-zip-stream directory.

<stream name="SampleHdfsZipLineStream" class="com.jkoolcloud.tnt4j.streams.inputs.HdfsZipLineStream">
    <property name="FileName"
              value="hdfs://[host]:[port]/[path]/sample.zip!2/*.txt"/>
    <property name="ArchType" value="ZIP"/>
    <.../>
</stream>

To stream HDFS zipped file lines HdfsZipLineStream shall be used. FileName is defined using URI starting hdfs://.