Recommendations for handling increased throughput in high-volume systems

The following recommendations are suggestions for changes that you can make if your system is producing a high volume of logs.

1. Optimize log messages to avoid logging unused tnt4j information. In the tnt4j.properties file, set the following:

     event.formatter: com.jkoolcloud.tnt4j.format.DefaultFormatter
     event.formatter.Format: {2}

2. Use random access files. In the log4j2.xml file:
   1. Change from RollingFile to RollingRandomAccessFile.
   2. Add the following attribute to RollingRandomAccessFile: immediateFlush="false"
3. Enable asynchronous loggers. This requires the disruptor library disruptor-3.4.4.jar. AutoPilot versions 6.0.34.6 and later include this jar file. Random access file must be used. Asynchronous logging can be set:

• Via system property:

     -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector

• Via system property:

     -Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.BasicAsyncLoggerContextSelector

• Via log4j2.xml file:
  1. 1. 1. Change Logger to AsyncLogger
        2. Change Root to AsyncRoot

4. Filter by level on the tnt4j side. In the tnt4j.properties file:

    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: INFO

5. Remove duplicated messages. In the tnt4j.properties file:

    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: INFO
    event.sink.factory.Filter.SuppressDups: true
    event.sink.factory.Filter.SuppressUseSoundex: false
    event.sink.factory.Filter.SuppressTimeSec: 2
    event.sink.factory.Filter.SuppressCacheSize: 500

6. Remove more duplicated messages using Soundex (which codes strings based on similar phonemes). In the tnt4j.properties file:

    event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
    event.sink.factory.Filter.Level: INFO
    event.sink.factory.Filter.SuppressDups: true
    event.sink.factory.Filter.SuppressUseSoundex: true
    event.sink.factory.Filter.SuppressTimeSec: 2
    event.sink.factory.Filter.SuppressCacheSize: 500

7. Additional asynchronous logger configurations can be set via system properties or in the log4j2.component.properties file. For more information, see the following documentation:
   1. • https://logging.apache.org/log4j/2.x/manual/async.html
      • https://logging.apache.org/log4j/2.x/manual/configuration.html#SystemProperties

For example, when a message queue is full, you can discard all messages with a severity less than WARN:

   log4j2.AsyncQueueFullPolicy=Discard
   log4j2.discardThreshold=WARN