How can I save logs from all meshIQ components to a centralized or user-defined path? – meshIQ

This article covers following components.

AutoPilotM6 logs
Stream JMX for Kafka & Confluent monitoring
Connection Managers or Agent
Other Components

1. AutoPilotM6 logs

The configuration file, log4j2.xml, located under $AUTOPILOT_HOME (which can be /opt/nastel/AutoPilotM6, /opt/meshiq/platform, or any of the user installed paths), manages the log file path. The different types of AutoPilot logs and how they can be redirected to a user required path is listed below.

Domain logs
All the information written to this log is related to Domain Server's tasks, responsibilities, and the services deployed on it.
In log4j2.xml file, the configuration for Domain logs is defined within the RollingFile name="file". The path on the disk, where the log file is created is defined between the same RollingFile tag.
For example, if we want to save logs in the /logs/AutoPilotM6/logs/log4j directory, log4j2.xml configuration would look as follows.
```
<RollingFile name="file"
             fileName="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}.log4j"
             filePattern="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}.%i.log4j.gz"
             ignoreExceptions="false">
```
If we also have the configuration related to storing logs for certain days, then <DefaultRolloverStrategy should be updated as shown below.
```
<DefaultRolloverStrategy max="20" fileIndex="min">
    <Delete basePath="/logs/AutoPilotM6/logs/log4j" maxDepth="2">
        <IfFileName glob="${sys:server.container.name}*.log4j.gz"/>
        <IfLastModified age="P7D" />
    </Delete>
</DefaultRolloverStrategy>
```
CEP logs
Using this log, end users get information on the reasons or causes for operation failures related to the services deployed on CEP. CEP logs are also controlled using same tag
RollingFile name="file". Therefore, the changes mentioned above will not only affect the Domain logs but also the CEP logs, saving them in a user-defined path.
Workgroup Server logs
The Workgroup Server log is generated by WGS component. It allows user to debug problems related WGS startup and any failure in WGS operations. In the log4j2.xml file, the configuration for WGS logs is defined in RollingFile name="file_WGSExpert". The path on the disk, where the log file is created, is defined between the same RollingFile tag.
For example, if we want to save our logs in the /logs/AutoPilotM6/logs/log4j directory, log4j2.xml configuration would look as follows.
```
<RollingFile name="file_WGSExpert"
             fileName="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}_wgs.log4j"
             filePattern="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}_wgs.%i.log4j.gz"
             ignoreExceptions="false">
```
If we also have the configuration related to storing logs for a certain number of days, then <DefaultRolloverStrategy should be updated as shown below.
```
<DefaultRolloverStrategy max="20" fileIndex="min">
    <Delete basePath="/logs/AutoPilotM6/logs/log4j" maxDepth="2">
        <IfFileName glob="*_wgs.*.log4j.gz"/>
        <IfLastModified age="P7D" />
    </Delete>
</DefaultRolloverStrategy>
```
Event logs
Event logs, as the name suggests, store a list of events that occurred on policies, policy states, or deployed services, and other related activities. To change the event log file location, add below property to global.properties file.
```
property server.log.dir=/logs/AutoPilotM6/logs
```
If the user wants to update the size of event logs, it can be altered using following property. Note that the value is in bytes.
```
property server.log.size = 500000
```

2. Stream JMX for Kafka & Confluent monitoring

Stream JMX logs are also managed using log4j2.xml, located at $STREAM_JMX_HOME/config (typically /opt/nastel/tnt4j-stream-jmx/current/config or /opt/meshiq/stream-jmx/current/config). These logs provide insights into JMX connection issues and problems with publishing collected JMX metrics to the endpoint.

For stream-jmx version 1.X or 11.X, the following properties should be updated in the log4j2.xml file. We will assume that the path to save logs is /logs/stream-jmx/:

<Property name="tnt4j.stream.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_${sys:sjmx.serviceId}.log</Property>
<Property name="tnt4j.activities.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_samples_${sys:sjmx.serviceId}.log</Property>

For stream-jmx version 12.X, the following properties should be updated in the log4j2.xml file. We will assume that the path to save logs is /logs/stream-jmx/:

<Property name="tnt4j.stream.log.dir">/logs/stream-jmx</Property>
<Property name="tnt4j.stream.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_${sys:sjmx.serviceId}.log</Property>
<Property name="tnt4j.activities.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_samples_${sys:sjmx.serviceId}.log}</Property>

3. Connection Managers or Agent

All Java connection managers (nsqcmmq, nsqcmems, nsqcmkafka, nsqcmace, nsqcmsolace, nsqcmrabbitmq) have their logging configurations defined in log4j2.xml. For C executables, such as MQ Agent (nsqmq), Publish-Subscribe (nsqpub), Message Server (nsqmsg), or Connection Manager (nsqcm), the logging path is typically defined in the startup script.

Java Connection Managers (nsqcmmq, nsqcmems, nsqcmkafka, nsqcmace, nsqcmsolace, nsqcmrabbitmq)

For Java Connection Managers, the log4j2.xml is packaged with the jar file, and the path to save log file is pre-defined. For this example, we will considering the Kafka CM. To save log file to a different path, you need to create a new log4j2.xml file under the $CM_HOME directory (e.g., /opt/nastel/apwmq/nsqcmkafka) with the contents shown below. In the example below, we assume the log file path is /logs/apwmq/kafkacm/NameOfTheNode.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR">
<Appenders>
<Console name="LogToConsole" target="SYSTEM_OUT">
<PatternLayout>
<Pattern>%d{ISO8601} %5p [%t] (%c{1}) - %m%n</Pattern>
</PatternLayout>
</Console>
<RollingRandomAccessFile
name="KafkaLogToRollingRandomAccessFile" fileName="/logs/apwmq/kafkacm/nsqcmkafka/nsqcmkafka.log4j"
filePattern="/logs/apwmq/kafkacm/nsqcmkafka/nsqcmkafka.log4j.%i">
<PatternLayout>
<Pattern>%d{ISO8601} %-5p[%c{1}][%t] - %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
<DefaultRolloverStrategy max="3" />
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<AsyncLogger name="com.nastel.kafka" level="TRACE"
additivity="false">
<AppenderRef ref="KafkaLogToRollingRandomAccessFile" />
<AppenderRef ref="LogToConsole" />
</AsyncLogger>
<AsyncLogger name="WGSExpert" level="TRACE"
additivity="false">
<AppenderRef ref="KafkaLogToRollingRandomAccessFile" />
<AppenderRef ref="LogToConsole" />
</AsyncLogger>
<AsyncRoot level="ERROR">
<AppenderRef ref="LogToConsole" />
</AsyncRoot>
</Loggers>
</Configuration>

Once the log4j2.xml file is created under the $CM_HOME directory, you need to define the log4j2.xml file that application should use with the startup parameters. For example, it would look like:

java -Dlog4j.configurationFile=/opt/nastel/apwmq/nsqcmkafka/log4j2.xml -jar $KAFKA_CM_HOME/$KAFKAEXE_NAME >/dev/null 2>&1 &

MQ Connection manager (nsqcm)
As previously stated, the MQ connection manager log file is defined in the startup script. This start script is either located at /opt/nastel/apwmq/sbin/nsqcm_start.sh or /opt/nastel/sbin/wmq/nsqcm_start.sh. The line that controls or sends output from nsqcm to the log file is shown below. In this example, we're logging to the /logs/apwmq/mqcm/ path.
```
nsqcm -mmqm -nREMOTE_QMGRS -s5011 -u >> /logs/apwmq/MQCM/nsqcm.log 2>&1 &
```
Note that above log will not roll over and will continue to grow. It is recommended to use nsqcirclog utility, that would roll logs based on the way user defines and we would not go into out of disk issues. Using nsqcirclog is a preferable approach. This utility is located at $APWMQ_HOME/bin (typically, /opt/nastel/apwmq/bin or /opt/meshiq/manage/apwmq/bin). Refer to resource centre article, nsqcirclog Utility – meshIQ for understanding the usage of nsqcirclog.
```
nsqcm -mmqm -nREMOTE_QMGRS -s5011 -u | $APWMQ_HOME/bin/nsqcirclog -ln 10K -fn 10 -filename /logs/apwmq/MQCM/nsqcm.log 2>&1 &
```
MQ Agent (nsqmq), Pub-Sub (nsqpub) or Message Server (nsqmsg)
Similar to the MQ connection manager, for the MQ Agent (nsqmq), Publish-Subscribe (nsqpub), or Message Server (nsqmsg), we will update the startup scripts located at the same location. The startup script names are nsqmq_start.sh, nsqmsg_start.sh and nsqpub_start.sh.

4. Other Components

This section covers additional concepts such as Syslog Event Monitor or policy logging to a file.

Syslog Event Monitor
The Syslog expert listens on a port and saves the solace events or commands to log file. This configuration is present in Syslog Options tab. See the screenshot below. If you want to save the Syslogs to a separate directory, the Syslog filename field should be updated to /logs/syslog/syslog.log.
Policy Logging to File

Policy logging to a file can be done in two ways:
1. User Action: The logging path is defined in the script that collects alert information and processes it to a log file. In this case, if you want to change the logging path, you need to modify the script.
2. Policy/Sensor Properties (Logging Tab): The path to the log file is defined using the Log file field. See the screenshot attached below, which shows how to configure the log file path.

1. AutoPilotM6 logs

Domain logs

CEP logs

Workgroup Server logs

Event logs

2. Stream JMX for Kafka & Confluent monitoring

3. Connection Managers or Agent

Java Connection Managers (nsqcmmq, nsqcmems, nsqcmkafka, nsqcmace, nsqcmsolace, nsqcmrabbitmq)

MQ Connection manager (nsqcm)

MQ Agent (nsqmq), Pub-Sub (nsqpub) or Message Server (nsqmsg)

4. Other Components

Syslog Event Monitor

Policy Logging to File