Recommendations for handling increased throughput in high-volume systems

The following recommendations are suggestions for changes that you can make if your system is producing a high volume of logs.

1. Optimize log messages to avoid logging unused tnt4j information. In the tnt4j.properties file, find the stanza, whose source has value as source: * and set the following (if property is already there, just update the value as per requirement):

event.formatter: com.jkoolcloud.tnt4j.format.DefaultFormatter
event.formatter.Format: {2}

2. In the log4j2.xml file:
   • Change all the occurrences of RollingFile to RollingRandomAccessFile.
   • Append/Add the immediateFlush="false" attribute to RollingRandomAccessFile section.
   • Change all the occurrences of Logger to AsyncLogger. Please note that this change is only to Logger and not required for Loggers.
   • Change all the occurrences of Root  to AsyncRoot
3. Enable asynchronous loggers. This requires the disruptor library disruptor-X.X.X.jar to be present under $AUTOPILOT_HOME/lib. AutoPilot versions 6.0.34.6 and later include this jar file. Random access file must be used. Asynchronous logging can be set via either of the system properties listed below (note that values or classes used are different). These system properties can be defined in each of the components config file, like
   • ATPNAMES.lax for Domain component. located at $AUTOPILOT_HOME/naming.
   • ATPNODE.lax for CEP component. Located at $AUTOPILOT_HOME/localhost.
   • ATPCONS.lax for EM component. Located at $AUTOPILOT_HOME/mconsole.
   • Any other component, like CM needs to be defined in either startup parameters or either at OS level.

System property to be used can be:

-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector

Or

-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.BasicAsyncLoggerContextSelector

4. Filter by level on the tnt4j side. In the tnt4j.properties file, set filter level to INFO. Sample attached below.

event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: INFO

5. If there are any duplicate entries in each section of tnt4j.properties file, remove them. For example:

event.sink.factory.Filter: com.jkoolcloud.tnt4j.filters.EventLevelTimeFilter
event.sink.factory.Filter.Level: INFO
event.sink.factory.Filter.SuppressDups: true
event.sink.factory.Filter.SuppressUseSoundex: false
event.sink.factory.Filter.SuppressTimeSec: 2
event.sink.factory.Filter.SuppressCacheSize: 500

6. Additional asynchronous logger configurations can be set via system properties or in the log4j2.component.properties file. For more information, see the following documentation:
   1.  
      • https://logging.apache.org/log4j/2.x/manual/async.html
      • https://logging.apache.org/log4j/2.x/manual/configuration.html#SystemProperties

For example, when a message queue is full, you can discard all messages with a severity less than WARN:

log4j2.AsyncQueueFullPolicy=Discard
log4j2.discardThreshold=WARN