TNT4J Streams for handling Syslog messages.

TNT4J-Streams-Syslogd is under Nastel Technologies EULA.

TNT4J-Streams-Syslogd is an extension of TNT4J-Streams that provides the ability to stream Syslog events/log entries as activity events to XRay/Track.

This document covers only information specific to the TNT4J-Streams-Syslogd project. Detailed information on TNT4J-Streams can be found in TNT4J-Streams.

• Allows streaming activities parsed from Syslog daemon (Syslogd) events data.
• Allows streaming activities parsed from Syslog log files.

Currently supports RFC 3164 and the Structured Syslog protocol RFC 5424.

• Select File>Import...>Maven>Existing Maven Projects
• Click Next.
• In 'Root directory' field select path of directory where you have downloaded (checked out from git) TNT4J-Streams project.
• Click OK.
• The dialog fills in with project modules details.
• Click Finish.

See TNT4J-Stream document in TNT4J-Streams for more details.

• As a standalone application
  • Write a streams configuration file. See Streams configuration for more details.
  • Configure your loggers.
  • Use bin/tnt4j-streams.bat or bin/tnt4j-streams.sh to run the standalone application.
• As an API integrated into your product

  • Use the Maven dependency:

    <dependency>
        <groupId>com.jkoolcloud.tnt4j.streams</groupId>
        <artifactId>tnt4j-streams-syslogd</artifactId>
        <version>12.0.0</version>
    </dependency>

  • Write a streams configuration file.  See Streams configuration for more details
  • Use StreamsAgent.runFromAPI(new CfgStreamsBuilder().setConfig(configFileName)) in your code.

When release assemblies are built, samples are located in samples directory, e.g., build/tnt4j-streams-syslogd-12.0.0/samples. To run desired sample:

• Go to the sample directory.
• Run run.bat or run.sh depending on your OS.

For more detailed explanation of streams and parsers configuration and usage, see Configuring TNT4J-Streams-Syslogd and JavaDocs.

This sample shows how to stream activity events from Syslogd received log events data. SyslogdStream starts Syslogd server depending on defined configuration.

See Syslog daemon (Syslogd) sample for more details.

This sample shows how to stream activity events from Syslog log file(s) entries.

See Syslog Log File sample for more details.

Details on TNT4J-Streams related configuration Configuring TNT4J-Streams.

• Host - host name to run Syslog server. (Required)
• Port - port number to run Syslog server. Default value - 514. (Optional)
• Protocol - protocol of Syslog server communication: one of tcp or udp . Default value - tcp. (Optional)
• Timeout - server communication timeout, where 0 means - server implementation dependent timeout handling. Actual if Protocol property value is set to tcp. Default value - 0. (Optional)

Configuration sample:

<property name="Protocol" value="udp"/>
<property name="Host" value="0.0.0.0"/>
<property name="Port" value="5114"/>
<property name="Timeout" value="60"/>

Also see Generic streams parameters and Buffered streams parameters.

• SuppressMessagesLevel - Syslog messages suppression level:

  • 0 - output all Syslog messages
  • -1 - output only the first occurrence of Syslog message
  • any other positive number - suppresses all Syslog messages except those that are multiples of that number

  Default value - 0. (Optional)

• SuppressIgnoredFields - Syslog message ignored fields list used to compare if message contents are same. Default value - [EndTime, ElapsedTime, Tag]. (Optional)

• SuppressCacheSize - maximal Syslog messages suppression cache entries count. Default value - 100. (Optional)

• SuppressCacheExpireDurationMinutes - Syslog messages suppression cache entries expiration duration value in minutes. Default value - 10. (Optional)

• FlattenStructuredData - flag indicating to flatten structured data map if there is only one structure available. Default value - false. (Optional)

Configuration sample:

<property name="SuppressMessagesLevel" value="10"/>
<property name="SuppressIgnoredFields" value="EndTime|ElapsedTime|Tag"/>
<property name="SuppressCacheSize" value="1000"/>
<property name="SuppressCacheExpireDurationMinutes" value="30"/>
<property name="FlattenStructuredData" value="true"/>

Also see Activity Map Parser.

This parser has no additional configuration parameters.

Also see Abstract Syslog Parser.

This parser resolves a data map that may contain such entries:

• For activity fields:

  • EventType - resolved from log event application message contained variable named opt
  • EventName - resolved log event facility name, or application message contained variable named opn
  • Exception - resolved from log event application message contained variable named exc
  • UserName - resolved from log event application message contained variable named usr
  • ResourceName - resolved log event application name, or application message contained variable named rsn
  • Location - resolved log event host name/address, or application message contained variable named loc
  • Tag - resolved set of values {host name, application name} for RFC 3164 and set of values {facility name, host name, application name, message id} for RFC 5424, or application message contained variable named tag
  • Correlator - resolved from log event application message contained variable named cid
  • ProcessId - resolved log event process id
  • ThreadId - same as ProcessId
  • Message - resolved log event application message
  • Severity - resolved log event level mapped to value from OpLevel enumeration
  • ApplName - resolved log event application name`
  • ServerName - resolved log event host name
  • EndTime - resolved log event timestamp value in microseconds
  • ElapsedTime - calculated time difference between same host and app events in microseconds
  • MsgCharSet - resolved log event char set name

• For activity properties:

  • facility - resolved log event facility name
  • level - resolved log event level
  • hostname - resolved log event host name
  • hostaddr - resolved log event host address
  • priority - resolved log line priority

• Maps of resolved additional custom activity properties:

  • SyslogMap - map of resolved RFC 5424 structured data: contains sub-map for every found structure, but can be flattened to single level map (if only one structure is available) using parser property FlattenStructuredData
  • SyslogVars - a map of resolved application message variables (varName=varValue) 

• CharSet - name of char set used by Syslog lines parser. Default value - UTF-8. (Optional)

Configuration sample:

<property name="CharSet" value="ISO-8859-1"/>

Also see Abstract Syslog Parser.

This parser resolves a data map that may contain such entries:

• For activity fields:

  • EventType - resolved from log line application message contained variable named opt
  • EventName - resolved log line facility name, or application message contained variable named opn
  • Exception - resolved from log line application message contained variable named exc
  • UserName - resolved from log line application message contained variable named usr
  • ResourceName - resolved log line application name, or application message contained variable named rsn
  • Location - resolved log line host name/address, or application message contained variable named loc
  • Tag - resolved set of values {host name, application name} for RFC 3164 and set of values {facility name, host name, application name, message id} for RFC 5424, or application message contained variable named tag
  • Correlator - resolved from log line application message contained variable named cid
  • ProcessId - resolved log line process id
  • ThreadId - same as ProcessId
  • Message - resolved log line application message
  • Severity - resolved log line level mapped to value from OpLevel enumeration
  • ApplName - resolved log line application name
  • ServerName - resolved log line host name
  • EndTime - resolved log line timestamp value in microseconds
  • ElapsedTime - calculated time difference between same host and app events in microseconds
  • MsgCharSet - char set name used by parser

• For activity properties:

  • facility - resolved log line facility name. If resolved priority is null - then value is user
  • level - resolved log line level. If resolved priority is null - then value is INFO
  • hostname - resolved log line host name
  • version - resolved log line Syslog version (0 for RFC 3164, 1 for RFC 5424)
  • priority - resolved log line priority

• Maps of resolved additional custom activity properties:

  • SyslogMap - map of resolved RFC 5424 structured data: contains sub-map for every found structure, but can be flattened to single level map (if only one structure is available) using parser property FlattenStructuredData
  • SyslogVars - map of resolved application message contained varName=varValue variables

• JDK 17+
• Apache Maven 3
• TNT4J-Streams core module in particular

All other required dependencies are defined in project pom.xml file. If Maven is running online mode it should download these defined dependencies automatically.

If you have built and installed TNT4J-Streams into your local Maven repository, you don't need to install it manually.

TNT4J-Streams-Syslogd project does not require any manually downloaded dependencies at the moment.

Also see TNT4J-Streams Manually Installed Dependencies.

• To build the project, run Maven goals clean package
• To build the project and install to local repo, run Maven goals clean install
• To make distributable release assemblies use one of profiles: pack-assembly and pack-sources-and-javadocs:
  • Containing only binary (including test package) distribution: run mvn -P pack-assembly
  • Containing binary (including test package), source and javadoc distribution: run mvn -P pack-sources-and-javadocs,pack-assembly
• To make maven required source and javadoc packages, use profile pack-sources-and-javadocs
• To make maven central compliant release having source, javadoc and all signed packages, use final-release profile

Release assemblies are built to build/ directory.

Sometimes Maven fails to correctly handle dependencies. If dependency configuration looks fine, but Maven still complains about missing dependencies try to delete local Maven repository by hand: e.g., on MS Windows delete contents of c:\Users\[username]\.m2\repository directory.

So, to summarize, the quick "how to build" steps are as follows:

1. If tnt4j-streams has not been built yet, build it: run mvn clean install for the pom.xml file located in the tnt4j-streams directory.
2. Now you can build tnt4j-streams-syslogd: run mvn clean install for a pom.xml file located in tnt4j-streams-syslogd directory.

See Running TNT4J-Streams-Syslogd samples.

• JUnit 4
• Mockito

Maven tests run is disabled by default. To enable Maven to run tests set Maven command line argument -DskipTests=false or use profile run-tests.

In the syslogd module, run the JUnit test suite named AllSyslogdStreamTests.