How can I save logs from all meshIQ components to a centralized or user-defined path? – meshIQ

This article covers following components.

AutoPilotM6 logs
Tomcat logs
Stream JMX for Kafka & Confluent monitoring
Connection Managers or Agent
Zookeeper (Track/Xray)
Solr (Track/Xray)
ActiveMQ (Track/Xray)
Kafka (Track/Xray)
Other Components

It is recommended to take backup of entire file, before editing it. Backup can be used to revert configuration, in case of requirement.

1. AutoPilotM6 logs

The configuration file, log4j2.xml, located under $AUTOPILOT_HOME (which can be /opt/nastel/AutoPilotM6, /opt/meshiq/platform, or any of the user installed paths), manages the log file path. The different types of AutoPilot logs and how they can be redirected to a user required path is listed below.

Domain logs
All the information written to this log is related to Domain Server's tasks, responsibilities, and the services deployed on it.
In log4j2.xml file, the configuration for Domain logs is defined within the RollingFile name="file". The path on the disk, where the log file is created is defined between the same RollingFile tag.
For example, if we want to save logs in the /logs/AutoPilotM6/logs/log4j directory, log4j2.xml configuration would look as follows.
```
<RollingFile name="file"
             fileName="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}.log4j"
             filePattern="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}.%i.log4j.gz"
             ignoreExceptions="false">
```
If we also have the configuration related to storing logs for certain days, then <DefaultRolloverStrategy should be updated as shown below.
```
<DefaultRolloverStrategy max="20" fileIndex="min">
    <Delete basePath="/logs/AutoPilotM6/logs/log4j" maxDepth="2">
        <IfFileName glob="${sys:server.container.name}*.log4j.gz"/>
        <IfLastModified age="P7D" />
    </Delete>
</DefaultRolloverStrategy>
```
CEP logs
Using this log, end users get information on the reasons or causes for operation failures related to the services deployed on CEP. CEP logs are also controlled using same tag
RollingFile name="file". Therefore, the changes mentioned above will not only affect the Domain logs but also the CEP logs, saving them in a user-defined path.
Workgroup Server logs
The Workgroup Server log is generated by WGS component. It allows user to debug problems related WGS startup and any failure in WGS operations. In the log4j2.xml file, the configuration for WGS logs is defined in RollingFile name="file_WGSExpert". The path on the disk, where the log file is created, is defined between the same RollingFile tag.
For example, if we want to save our logs in the /logs/AutoPilotM6/logs/log4j directory, log4j2.xml configuration would look as follows.
```
<RollingFile name="file_WGSExpert"
             fileName="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}_wgs.log4j"
             filePattern="/logs/AutoPilotM6/logs/log4j/${sys:server.container.name}_wgs.%i.log4j.gz"
             ignoreExceptions="false">
```
If we also have the configuration related to storing logs for a certain number of days, then <DefaultRolloverStrategy should be updated as shown below.
```
<DefaultRolloverStrategy max="20" fileIndex="min">
    <Delete basePath="/logs/AutoPilotM6/logs/log4j" maxDepth="2">
        <IfFileName glob="*_wgs.*.log4j.gz"/>
        <IfLastModified age="P7D" />
    </Delete>
</DefaultRolloverStrategy>
```
Event logs
Event logs, as the name suggests, store a list of events that occurred on policies, policy states, or deployed services, and other related activities. To change the event log file location, add below property to global.properties file.
```
property server.log.dir=/logs/AutoPilotM6/logs/
```
If the user wants to update the size of event logs, it can be altered using following property. Note that the value is in bytes.
```
property server.log.size = 500000
```

2. Tomcat logs

These instruction apply to Tomcat 9.x configuration.

Tomcat log path is defined in more than one location.

In $CATALINA_HOME/conf directory
(e.g., /opt/nastel/AutoPilotM6/apache-tomcat/conf or /opt/meshiq/platform/apache-tomcat/conf),
open the logging.properties file and update the path for all properties ending with .directory.
The list of properties to update is shown below. In this example, logs are saved to the path /logs/tomcat:
```
1. catalina.org.apache.juli.AsyncFileHandler.directory = /logs/tomcat
2. localhost.org.apache.juli.AsyncFileHandler.directory = /logs/tomcat
3. manager.org.apache.juli.AsyncFileHandler.directory = /logs/tomcat
4. host-manager.org.apache.juli.AsyncFileHandler.directory = /logs/tomcat
```
In the $CATALINA_HOME/bin directory
(e.g., /opt/nastel/AutoPilotM6/apache-tomcat/bin or /opt/meshiq/platform/apache-tomcat/bin),

open the catalina.sh file, search for the environment variable CATALINA_OUT, and update its path as shown below:
```
CATALINA_OUT=/logs/tomcat/catalina.out
```
To configure logging for meshIQ web applications (such as Navigator/Manage and XRay/Track), go to:
$CATALINA_HOME/webapps/$WEBAPPDIR/WEB-INF/classes

Possible paths include:
- /opt/nastel/AutoPilotM6/apache-tomcat/webapps/navigator-server/WEB-INF/classes
- /opt/nastel/AutoPilotM6/apache-tomcat/webapps/xray/WEB-INF/classes
- /opt/meshiq/platform/apache-tomcat/webapps/manage-server/WEB-INF/classes
- /opt/meshiq/platform/apache-tomcat/webapps/track/WEB-INF/classes
Open the log4j2.xml file and update the fileName and filePattern fields as shown below:
```
name="LogToRollingRandomAccessFile" fileName="/logs/tomcat/navigator.log"
filePattern="/logs/tomcat/navigator.%d{yyyy-MM-dd}.log">
```

3. Stream JMX for Kafka & Confluent monitoring

Stream JMX logs are also managed using the log4j2.xml file located at $STREAM_JMX_HOME/config (typically found at /opt/nastel/tnt4j-stream-jmx/current/config or /opt/meshiq/stream-jmx/current/config).
These logs provide insights into JMX connection issues and problems related to publishing collected JMX metrics to the endpoint.

For stream-jmx version 1.X or 11.X, update the following properties in the log4j2.xml file. Assume the log path is /logs/stream-jmx/:

<Property name="tnt4j.stream.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_${sys:sjmx.serviceId}.log</Property>
<Property name="tnt4j.activities.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_samples_${sys:sjmx.serviceId}.log</Property>

For stream-jmx version 12.X, update the following properties in the log4j2.xml file, also assuming the log path is /logs/stream-jmx/:

<Property name="tnt4j.stream.log.dir">/logs/stream-jmx</Property>
<Property name="tnt4j.stream.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_${sys:sjmx.serviceId}.log</Property>
<Property name="tnt4j.activities.log.filename">/logs/stream-jmx/${sys:sjmx.serviceId}/tnt4j-stream-jmx_samples_${sys:sjmx.serviceId}.log}</Property>

Stream JMX start-up scripts also write logs to a file, with the path defined in the script using an environment variable. Update it as shown below:

export STREAM_JMX_LOGS="/logs/stream-jmx"

4. Connection Managers or Agent

All Java connection managers (nsqcmmq, nsqcmems, nsqcmkafka, nsqcmace, nsqcmsolace, nsqcmrabbitmq) have their logging configurations defined in log4j2.xml. For C executables, such as MQ Agent (nsqmq), Publish-Subscribe (nsqpub), Message Server (nsqmsg), or Connection Manager (nsqcm), the logging path is typically defined in the startup script.

Java Connection Managers (nsqcmmq, nsqcmems, nsqcmkafka, nsqcmace, nsqcmsolace, nsqcmrabbitmq)

For Java Connection Managers, the log4j2.xml is packaged with the jar file, and the path to save log file is pre-defined. To save log file to a different path, you need to create a new log4j2.xml file under the $CM_HOME directory (e.g., /opt/nastel/apwmq/nsqcmkafka) with the contents shown below. As there is a connection manager per middleware technology, attaching sample for each connection manager component separately below.

For nsqcmmq component, log4j2.xml would be as follows. In the example below, we assume the log4j path is /logs/apwmq/mqcm/NameOfTheNode.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="LogToConsole" target="SYSTEM_OUT">
<PatternLayout>
<Pattern>%d{ISO8601} %5p [%t] (%c{1}) - %m%n</Pattern>
</PatternLayout>
</Console>
<RollingRandomAccessFile
name="LogToRollingRandomAccessFile" fileName="./nsqcmmq.log4j"
filePattern="./nsqcmmq.log4j.%i">
<PatternLayout>
<Pattern>%d{ISO8601} %-5p[%c{1}][%t] - %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
<DefaultRolloverStrategy max="3" />
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<AsyncLogger name="com.nastel.agent" level="INFO" additivity="false">
<AppenderRef ref="LogToRollingRandomAccessFile" />
<AppenderRef ref="LogToConsole" />
</AsyncLogger>
<AsyncLogger name="WGSExpert" level="INFO" additivity="false">
<AppenderRef ref="LogToRollingRandomAccessFile" />
<AppenderRef ref="LogToConsole" />
</AsyncLogger>
<AsyncRoot level="WARN">
<AppenderRef ref="LogToConsole" />
</AsyncRoot>
</Loggers>
</Configuration>

For nsqcmems component, log4j2.xml would be as follows. In the example below, we assume the log4j path is /logs/apwmq/emscm/NameOfTheNode.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
 <Appenders>
 <Console name="LogToConsole" target="SYSTEM_OUT">
 <PatternLayout>
 <Pattern>%d{ISO8601} %5p [%t] (%c{1}) - %m%n</Pattern>
 </PatternLayout>
 </Console>
 <RollingRandomAccessFile
 name="LogToRollingRandomAccessFile" fileName="/logs/apwmq/emscm/nsqcmems/nsqcmems.log4j"
 filePattern="/logs/apwmq/emscm/nsqcmems/nsqcmems.log4j.%i">
 <PatternLayout>
 <Pattern>%d{ISO8601} %-5p[%c{1}][%t] - %m%n</Pattern>
 </PatternLayout>
 <Policies>
 <SizeBasedTriggeringPolicy size="10 MB" />
 </Policies>
 <DefaultRolloverStrategy max="3" />
 </RollingRandomAccessFile>
 </Appenders>
 <Loggers>
 <AsyncLogger name="com.nastel.ems" level="TRACE"
 additivity="false">
 <AppenderRef ref="LogToRollingRandomAccessFile" />
 <AppenderRef ref="LogToConsole" />
 </AsyncLogger>
 <AsyncLogger name="WGSExpert" level="TRACE"
 additivity="false">
 <AppenderRef ref="LogToRollingRandomAccessFile" />
 <AppenderRef ref="LogToConsole" />
 </AsyncLogger> 
 <AsyncRoot level="WARN">
 <AppenderRef ref="LogToConsole" />
 </AsyncRoot>
 </Loggers>
</Configuration>

For nsqcmkafka component, log4j2.xml would be as follows. In the example below, we assume the log file path is /logs/apwmq/kafkacm/NameOfTheNode.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR">
 <Appenders>
 <Console name="LogToConsole" target="SYSTEM_OUT">
 <PatternLayout>
 <Pattern>%d{ISO8601} %5p [%t] (%c{1}) - %m%n</Pattern>
 </PatternLayout>
 </Console>
 <RollingRandomAccessFile
 name="LogToRollingRandomAccessFile" fileName="/logs/apwmq/kafkacm/nsqcmkafka/nsqcmkafka.log4j"
 filePattern="/logs/apwmq/kafkacm/nsqcmkafka/nsqcmkafka.log4j.%i">
 <PatternLayout>
 <Pattern>%d{ISO8601} %-5p[%c{1}][%t] - %m%n</Pattern>
 </PatternLayout>
 <Policies>
 <SizeBasedTriggeringPolicy size="10 MB" />
 </Policies>
 <DefaultRolloverStrategy max="3" />
 </RollingRandomAccessFile>
 </Appenders>
 <Loggers>
 <AsyncLogger name="com.nastel.kafka" level="TRACE"
 additivity="false">
 <AppenderRef ref="LogToRollingRandomAccessFile" />
 <AppenderRef ref="LogToConsole" />
 </AsyncLogger>
 <AsyncLogger name="WGSExpert" level="TRACE"
 additivity="false">
 <AppenderRef ref="LogToRollingRandomAccessFile" />
 <AppenderRef ref="LogToConsole" />
 </AsyncLogger> 
 <AsyncRoot level="ERROR">
 <AppenderRef ref="LogToConsole" />
 </AsyncRoot>
 </Loggers>
</Configuration>

For nsqcmsolace component, log4j2.xml would be as follows. In the example below, we assume the log file path is /logs/apwmq/solacecm/NameOfTheNode.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
<Appenders>
<Console name="LogToConsole" target="SYSTEM_OUT">
<PatternLayout>
<Pattern>%d{ISO8601} %5p [%t] (%c{1}) - %m%n</Pattern>
</PatternLayout>
</Console>
<RollingRandomAccessFile
name="LogToRollingRandomAccessFile" fileName="/logs/apwmq/solacecm/nsqcmsolace/nsqcmsolace.log4j"
filePattern="/logs/apwmq/solacecm/nsqcmsolace/nsqcmsolace.log4j.%i">
<PatternLayout>
<Pattern>%d{ISO8601} %-5p[%c{1}][%t] - %m%n</Pattern>
</PatternLayout>
<Policies>
<SizeBasedTriggeringPolicy size="10 MB" />
</Policies>
<DefaultRolloverStrategy max="3" />
</RollingRandomAccessFile>
</Appenders>
<Loggers>
<AsyncLogger name="com.nastel.solace" level="TRACE"
additivity="false">
<AppenderRef ref="LogToRollingRandomAccessFile" />
<AppenderRef ref="LogToConsole" />
</AsyncLogger>
<AsyncLogger name="WGSExpert" level="TRACE"
additivity="false">
<AppenderRef ref="LogToRollingRandomAccessFile" />
<AppenderRef ref="LogToConsole" />
</AsyncLogger>
<AsyncRoot level="INFO">
<AppenderRef ref="LogToConsole" />
</AsyncRoot>
</Loggers>
</Configuration>

For nsqcmrabbitmq component, log4j2.xml file would be as follows. In the sample below, we're considering the log path as /logs/apwmq/rabbitmqcm/NameOfTheNode.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
 <Appenders>
 <Console name="LogToConsole" target="SYSTEM_OUT">
 <PatternLayout>
 <Pattern>%d{ISO8601} %5p [%t] (%c{1}) - %m%n</Pattern>
 </PatternLayout>
 </Console>
 <RollingRandomAccessFile
 name="LogToRollingRandomAccessFile" fileName="./nsqcmrabbitmq.log4j"
 filePattern="./nsqcmrabbitmq.log4j.%i">
 <PatternLayout>
 <Pattern>%d{ISO8601} %-5p[%c{1}][%t] - %m%n</Pattern>
 </PatternLayout>
 <Policies>
 <SizeBasedTriggeringPolicy size="10 MB" />
 </Policies>
 <DefaultRolloverStrategy max="3" />
 </RollingRandomAccessFile>
 </Appenders>
 <Loggers>
 <AsyncLogger name="com.nastel.rabbitmq" level="TRACE"
 additivity="false">
 <AppenderRef ref="LogToRollingRandomAccessFile" />
 <AppenderRef ref="LogToConsole" />
 </AsyncLogger>
 <AsyncLogger name="WGSExpert" level="TRACE"
 additivity="false">
 <AppenderRef ref="LogToRollingRandomAccessFile" />
 <AppenderRef ref="LogToConsole" />
 </AsyncLogger> 
 <AsyncRoot level="WARN">
 <AppenderRef ref="LogToConsole" />
 </AsyncRoot>
 </Loggers>
</Configuration>

Once the log4j2.xml file is created under the $CM_HOME directory, you need to define the log4j2.xml file that application should use with the startup parameters (which is start.sh script located at $APIN_HOME/sbin). For example, it would look like below for kafkacm. Changes should be made appropriately, for each of the CMs (in their methods).

java -Dlog4j.configurationFile=$KAFKA_CM_HOME/log4j2.xml -jar $KAFKA_CM_HOME/$KAFKAEXE_NAME >/dev/null 2>&1 &

MQ Connection manager (nsqcm)
As previously stated, the MQ connection manager log file is defined in the startup script. This start script is either located at /opt/nastel/apwmq/sbin/nsqcm_start.sh or /opt/nastel/sbin/wmq/nsqcm_start.sh. The line that controls or sends output from nsqcm to the log file is shown below. In this example, we're logging to the /logs/apwmq/mqcm/ path.
```
nsqcm -mMQM -nREMOTE_QMGRS -s5011 -u >> /logs/apwmq/MQCM/nsqcm.log 2>&1 &
```
Note that above log will not roll over and will continue to grow. It is recommended to use nsqcirclog utility, that would roll logs based on the way user defines and we would not go into out of disk issues. Using nsqcirclog is a preferable approach. This utility is located at $APWMQ_HOME/bin (typically, /opt/nastel/apwmq/bin or /opt/meshiq/manage/apwmq/bin). Refer to resource centre article, nsqcirclog Utility – meshIQ for understanding the usage of nsqcirclog.
```
nsqcm -mMQM -nREMOTE_QMGRS -s5011 -u | $APWMQ_HOME/bin/nsqcirclog -ln 10K -fn 10 -filename /logs/apwmq/MQCM/nsqcm.log 2>&1 &
```
MQ Agent (nsqmq), Pub-Sub (nsqpub) or Message Server (nsqmsg)
Similar to the MQ connection manager, for the MQ Agent (nsqmq), Publish-Subscribe (nsqpub), or Message Server (nsqmsg), we will update the startup scripts located at the same location. The startup script names are nsqmq_start.sh, nsqmsg_start.sh and nsqpub_start.sh.

5. Zookeeper (Track/Xray)

Zookeeper logging is defined in Zookeeper start script zkServer.sh, located at $ZOOKEEPER_HOME/bin, typically /opt/meshiq/track/zookeeper/current/bin or /opt/nastel/zookeeper/current/bin. In this file, add environment variable ZOO_LOG_DIR and value should be the required log path.

For reference, see the example below, where logs are written to /logs/zoo:

ZOO_LOG_DIR=/logs/zoo

6. Solr (Track/Xray)

Solr logs path is defined in solr.in.sh file, located at $SOLR_HOME/bin, typically /opt/meshiq/track/zookeeper/current/bin or /opt/nastel/zookeeper/current/bin. In this file, value of SOLR_LOGS_DIR environment variable should be updated to required path.

In the sample below, Solr logs are written to the /logs/solr/logs directory:

SOLR_LOGS_DIR="/logs/solr/logs"

7. ActiveMQ (Track/Xray)

Active MQ logs configuration is defined in the log4j2.properties file, located at $ACTIVEMQ_HOME/conf, typically /opt/meshiq/track/actmq/current/conf or /opt/nastel/actmq/current/conf. The properties that need to be updated and their corresponding values are shown below. In this example, logs are redirected to the /logs/actmq location:

appender.logfile.fileName=/logs/actmq/activemq.log
appender.logfile.filePattern=/logs/actmq/activemq.log.%i

8. Kafka (Track/Xray)

Kafka logs path is defined in the kafka-run-class.sh file, located at $KAFKA_HOME/bin, typically /opt/meshiq/track/kafka/current/bin or /opt/nastel/kafka/current/bin. The variable that should be updated is LOG_DIR. You can either update the existing path or comment it out and add a new one, as shown in the example below, where Kafka logs are written to /logs/kafka/logs:

LOG_DIR="/logs/kafka/logs"

9. Other Components

This section covers additional concepts such as Syslog Event Monitor or policy logging to a file.

Syslog Event Monitor
The Syslog expert listens on a port and saves the solace events or commands to log file. This configuration is present in Syslog Options tab. See the screenshot below. If you want to save the Syslogs to a separate directory, the Syslog filename field should be updated to /logs/syslog/syslog.log.
Policy Logging to File

Policy logging to a file can be done in two ways:

1. 1. User Action: The logging path is defined in the script that collects alert information and processes it to a log file. In this case, if you want to change the logging path, you need to modify the script.
  2. Policy/Sensor Properties (Logging Tab): The path to the log file is defined using the Log file field. See the screenshot attached below, which shows how to configure the log file path.
Postgres SQL Database (Shipped with Deployment Pack)

Postgres logging is controlled using the startup options, and which are defined in the start.sh script located at $APIN_HOME/sbin (typically /opt/meshiq/sbin or /opt/nastel/sbin. The current startup option is:

${APIN_HOME}/pgsql/bin/pg_ctl -D ${APIN_HOME}/pgsql/data -l ${APIN_HOME}/logs/postgres.log start

Update this startup option to:

${APIN_HOME}/pgsql/bin/pg_ctl -D ${APIN_HOME}/pgsql/data -l /logs/postgres/postgres.log start

Start Scripts (Shipped with Deployment Pack)

Scripts such as start.sh, stop.sh, and config.sh write their outputs to /opt/nastel/logs or /opt/meshiq/logs.
To change this log path, edit the apin_env.sh file located at /opt/nastel/sbin or /opt/meshiq/sbin.
Update the APIN_LOGS variable as shown below. In this example, logs are redirected to the /logs/sbin path:
```
export APIN_LOGS=$APIN_HOME/logs/sbin
```

1. AutoPilotM6 logs

Domain logs

CEP logs

Workgroup Server logs

Event logs

2. Tomcat logs

3. Stream JMX for Kafka & Confluent monitoring

4. Connection Managers or Agent

Java Connection Managers (nsqcmmq, nsqcmems, nsqcmkafka, nsqcmace, nsqcmsolace, nsqcmrabbitmq)

MQ Connection manager (nsqcm)

MQ Agent (nsqmq), Pub-Sub (nsqpub) or Message Server (nsqmsg)

5. Zookeeper (Track/Xray)

6. Solr (Track/Xray)

7. ActiveMQ (Track/Xray)

8. Kafka (Track/Xray)

9. Other Components

Syslog Event Monitor

Policy Logging to File

Postgres SQL Database (Shipped with Deployment Pack)

Start Scripts (Shipped with Deployment Pack)