This article explains how to configure collectd to send operating system (OS) metrics to meshIQ Observe using tnt4j-streams.

If you want to collect Kafka JMX metrics instead of operating system metrics, see Configure Stream-JMX for Kafka Monitoring.

Prerequisites

collectd is installed and running on the monitored host.
Download the required tnt4j-streams package.
The machine running tnt4j-streams is reachable from the collectd host.

Procedure

Complete the following steps to configure collectd and Stream to send OS metrics to Observe.

Install tnt4j-streams

Download the tnt4j-streams-<version>.tgz package and extract it to a directory of your choice.
```
tar -xzf tnt4j-streams-<version>.tgz
cd tnt4j-streams-<version>
```

The extracted directory contains:

Directory	Description
`bin/`	Shell scripts used to start tnt4j-streams.
`samples/`	Sample stream configurations for various sources, including os-metrics/collectd covered below.
`lib/`, `opt/`	Libraries required by tnt4j-streams.

Configure collectd

The collectd configuration file is located at /etc/collectd/collectd.conf or /etc/collectd/collectd.conf.d/.
Locate the following plugin and uncomment it:
```
LoadPlugin write_http
```
Configure the plugin to send metrics as JSON to the Stream endpoint.

Add the following configuration near the end of the file.
```
<Plugin write_http>
 <Node "tnt4j_http_endpoint">
   URL "http://your-stream-host:9595/collectd"
   Format "JSON"
   Metrics true
   Notifications false
   StoreRates true
 </Node>
</Plugin>
```
Replace your-stream-host and the port with the hostname (or IP address) and port where your Stream instance is running. By default, the Stream service uses port 9595
Restart the collectd service by running:
```
sudo systemctl restart collectd
```
Verify that collectd has started successfully by reviewing the service log for any startup or communication errors. Run the following command:
```
sudo journalctl -u collectd -n 50
```

Configure tnt4j-streams

The tnt-data-source.xml file, located in samples/os-metrics/collectd/tnt-data-source.xml, configures how the Stream application receives metrics from collectd and forwards them to Observe.

Unlike JMX monitoring, there are no include or exclude filters in this file. The metrics collected are determined by the collectd plugin configuration. Stream service receives all metrics sent by collectd through the write_http plugin and forwards them to Observe.

Use the sample configuration provided with the package and update the required values.

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">
   <resource-ref id="CollectdParsers" type="Parser" uri="collectd-parsers.xml"/>
   <stream name="CollectdStream" class="com.jkoolcloud.tnt4j.streams.inputs.HttpStream">
       <property name="HaltIfNoParser" value="false"/>
       <!-- This is the host:port this stream listens on - point collectd's write_http plugin
            (URL setting) at this same host:port. -->
       <!-- CHANGE ME (optional): uncomment and set if you want to bind to one specific interface -->
       <!--property name="Host" value="localhost"/-->
       <!-- CHANGE ME: port collectd's write_http plugin should point its URL at -->
       <property name="Port" value="9595"/>
       <property name="SendStreamStates" value="false"/>
       <property name="BuildSourceFQNFromStreamedData" value="true"/>
       <!-- SourceFQN is a free-form "#"-separated list of key=value pairs, later used for filtering.
            It's read right to left, so put the narrowest token first (left) and the widest last
            (right) - here, HOST (a single machine) comes before ENV (a whole environment). HOST is
            dynamic - it's whatever hostname collectd itself reports with every payload (its own
            Hostname setting or OS hostname), so ${VMHost} is resolved live per host. Note this is
            the hostname collectd reports about itself, not an address resolved from the HTTP
            connection. ENV is static - change it to whatever fits your environment.
            CHANGE ME: ENV=DEV - set to your environment name -->
       <property name="SourceFQN" value="HOST=${VMHost}#ENV=DEV"/>
       <parser-ref name="CollectdReqParser"/>
       <tnt4j-properties>
           <!-- CHANGE ME (optional): GENERIC=OS sets the category name. Use something more specific
                if you like, e.g. GENERIC=LINUX-OS -->
           <property name="source.factory.GENERIC" value="OS"/>
           <property name="source.factory.RootFQN" value="GENERIC=?"/>
           <!-- Keep this the same value as source.factory.GENERIC above. -->
           <property name="tracker.default.snapshot.category" value="OS"/>
           <!-- Sequence of sinks to broadcast stream produced activities. Values may be one or a combination of: file, meshiq -->
           <property name="event.sink.factory.SinkThreads" value="1"/>
           <property name="event.sink.factory.BroadcastSequence" value="meshiq"/>
           <!-- File sink - useful for local testing/troubleshooting -->
           <property name="event.sink.factory.EventSinkFactory.file" value="com.jkoolcloud.tnt4j.sink.impl.FileEventSinkFactory"/>
           <property name="event.sink.factory.EventSinkFactory.file.FileName" value="./logs/activity_sink_collectd.log"/>
           <property name="event.sink.factory.EventSinkFactory.file.Formatter" value="com.jkoolcloud.tnt4j.format.JSONFormatter"/>
           <!-- meshIQ sink - streams to meshIQ Data Services Gateway -->
           <property name="event.sink.factory.EventSinkFactory.meshiq" value="com.jkoolcloud.jesl.tnt4j.sink.JKCloudEventSinkFactory"/>
           <!-- CHANGE ME: meshIQ Data Services Gateway URL -->
           <property name="event.sink.factory.EventSinkFactory.meshiq.Url" value="http://YOUR-GATEWAY-HOST:6580/"/>
           <!-- CHANGE ME: your streaming token -->
           <property name="event.sink.factory.EventSinkFactory.meshiq.Token" value="YOUR-STREAMING-TOKEN"/>
           <property name="event.sink.factory.EventSinkFactory.meshiq.AckSends" value="true"/>
           <property name="event.sink.factory.EventSinkFactory.meshiq.ConnectionPoolSize" value="5"/>
           <property name="event.sink.factory.EventSinkFactory.meshiq.Formatter" value="com.jkoolcloud.tnt4j.format.JSONFormatter"/>
       </tnt4j-properties>
   </stream>
</tnt-data-source>

Update the following properties in the tnt-data-source.xml file:

Property	Required Changes
`Host`	(Optional) Uncomment and set this property only if you want the Stream service to listen on a specific network interface. By default, it listens on all available interfaces.
`Port`	Update this value only if you want the Stream service to listen on a different port. Note: Ensure that the same port is configured in collectd's `write_http` URL. The default port is 9595.
`SourceFQN`	Update `ENV=DEV` to match your environment. `HOST=${VMHost}` is populated automatically from the hostname reported by collectd.
`source.factory.GENERIC`	Specifies the resource category displayed in Observe. The default value is `OS`. (Optional) Change the default value `OS` to a more descriptive category. If you change this value, update `tracker.default.snapshot.category` to use the same value.
`tracker.default.snapshot.category`	Keep this value the same as `source.factory.GENERIC`.
`event.sink.factory.EventSinkFactory.meshiq.Url`	Replace with the hostname or IP address of your meshIQ Data Services Gateway.
event.sink.factory.EventSinkFactory.meshiq.Token	Replace with the streaming token used to authenticate with the Data Services Gateway.

SourceFQN uniquely identifies the monitored resource using key=value pairs separated by #. The resource type does not need to be specified here because source.factory.GENERIC already identifies the data as OS metrics.

Understand the Parser Configuration

The collectd-parsers.xml file, located in samples/os-metrics/collectd/collectd-parsers.xml, defines how the Stream application parses raw JSON payloads received from collectd into metrics. This file is referenced by the tnt-data-source.xml configuration.

No changes are required to this file. Since the collectd payload format is fixed, you can use the provided configuration as-is.

<?xml version="1.0" encoding="utf-8"?>
<tnt-data-source
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="https://meshiq-xsd.s3.amazonaws.com/streams/tnt-data-source.xsd">
   <parser name="CollectdStatsDataParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
       <field name="EventType" value="Metric"/>
       <field name="TableName" formattingPattern="collectd.{0}" transparent="true" value-type="@TableName">
           <field-locator id="PluginName" locator="plugin" locator-type="Label"/>
       </field>
       <field name="VMHost" locator="host" locator-type="Label" transparent="true"/>
       <field name="Type" locator="type" locator-type="Label" value-type="@Dimension" transparent="true"/>
       <field name="Instance" locator="plugin_instance" locator-type="Label" value-type="@Dimension"/>
       <field name="TypeInstance" locator="type_instance" locator-type="Label" value-type="@Dimension" emptyAsNull="true"
              transparent="true"/>
       <field name="MetricNamePrefix" value="" transparent="true">
           <field-transform lang="groovy"><![CDATA[
               ${Type} + (Utils.isEmpty(${TypeInstance}) ? "" : "." + ${TypeInstance})
           ]]></field-transform>
       </field>
       <field name="${MetricNamePrefix}.${FieldNameLoc}" locator="values" locator-type="Label" value-type="${ValueTypeLoc}" split="true">
           <field-locator id="FieldNameLoc" locator="dsnames" locator-type="Label"/>
           <field-locator id="ValueTypeLoc" locator="dstypes" locator-type="Label"/>
       </field>
       <field name="StartTime" locator="time" locator-type="Label" datatype="Timestamp" units="Seconds"/>
       <field name="Interval" locator="interval" locator-type="Label" datatype="Number" format="double" value-type="@Metadata"/>
       <field name="EventName" value="">
           <field-transform lang="groovy"><![CDATA[
               String eName = Utils.toCompactString(${@Dimension})
               Utils.isEmpty(eName) ? ${TableName} : eName;
           ]]></field-transform>
       </field>
       <field name="Guid" separator="::">
           <field-locator id="Source" locator="Source" locator-type="Trackable"/>
           <field-locator id="Dimensions" locator="EventName" locator-type="Activity"/>
           <field-locator id="Time" locator="StartTime" locator-type="Activity" datatype="Number" format="long"/>
       </field>
       <field name="AllUnmapped" locator="#" locator-type="Label" datatype="AsInput" value-type="@Metadata"/>
   </parser>
   <parser name="CollectdReqBodyParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityJsonParser">
       <property name="ActivityDelim" value="EOF"/>
       <field name="EventType" value="NOOP"/>
       <field name="@SplitRelatives@" value="true" transparent="true"/>
       <embedded-activity name="OSMetrics" locator="$" locator-type="Label">
           <parser-ref name="CollectdStatsDataParser" aggregation="Relate"/>
       </embedded-activity>
   </parser>
   <parser name="CollectdReqParser" class="com.jkoolcloud.tnt4j.streams.parsers.ActivityMapParser">
       <field name="Transport" locator="ActivityTransport" locator-type="Label" value-type="@Metadata"/>
       <field name="MsgBody" locator="ActivityData" locator-type="Label" transparent="true">
           <parser-ref name="CollectdReqBodyParser"/>
       </field>
   </parser>
</tnt-data-source>

No changes are required to this file. The following properties explain how metric data is organized.

Property	Description
`TableName`	Defines the metric type in the format `collectd.<plugin>`, such as `collectd.cpu`, `collectd.memory`, or `collectd.df`. Each collectd plugin that reports data creates a separate metric type in Observe.
`VMHost`	Uses the hostname reported by collectd. This value populates the `${VMHost}` placeholder in the `SourceFQN` property defined in `tnt-data-source.xml`. It is not stored as a metric dimension.
`Instance`	Uses the `plugin_instance` value reported by collectd and adds it as a metric dimension when available. For example, it may identify a CPU core, network interface, or disk. System-wide metrics that do not include a `plugin_instance` are stored without an instance dimension.
`Metric Name`	Generated by combining the collectd Type, TypeInstance, and the corresponding value name (`dsnames`) from the collectd payload. For example, metric names such as `cpu.system` and `if_octets.rx` are created automatically.

Start the Stream Service

The tnt4j-streams receives metrics sent by collectd over HTTP. Start tnt4j-streams before starting or restarting collectd. If collectd sends metrics before the tnt4j-streams is listening, those metrics cannot be received and will be lost for that reporting interval.

Start the tnt4j-streams from the extracted package directory by running:
```
./bin/tnt4j-streams.sh -f:samples/os-metrics/collectd/tnt-data-source.xml
```
The -f option specifies the path to the Stream configuration file.

When the tnt4j-streams starts successfully, it begins listening on the configured port (default 9595).
After tnt4j-streams is running, restart collectd:
```
sudo systemctl restart collectd
```
Restart collectd after enabling the write_http plugin and configuring it to send metrics to tnt4j-streams. For more information, see Configure collectd.

Verify Metrics in Observe

After collectd metrics start flowing into Observe, configure a Resource Group and Monitor to view and monitor the collected metrics.

Sign in to the meshIQ Observe web interface.
Create a Resource Group. Verify that the Category drop-down lists the category configured in the source.factory.GENERIC property of the tnt-data-source.xml file (for example, OS).
Select the required resource type. The Type drop-down lists the collectd.<plugin> values currently reporting metrics.
Configure the Resource Group filters.
Use the SourceFQN values (such as HOST and ENV) to filter the resources. These values are populated from the SourceFQN property configured in tnt-data-source.xml.
Create a Monitor for the selected Resource Group.
Select the required metrics. Observe automatically lists all metrics available for the selected resource. Metric names are generated from the collectd Type, TypeInstance, and dsname values (for example, cpu.wait.value).
Configure alert thresholds for the required metrics and save the Monitor.

For additional information about collectd monitoring with tnt4j-streams, see docs/tnt4j-streams.md in the extracted package, under the collectd performance metrics streaming section.