Skip to main content
Top
Review meshIQ Products (G2.com)

How do I set up an AWS CloudWatch metrics HTTP endpoint?

This article applies to XRay versions 1.5 and 1.6 and to the meshIQ Platform versions 11 and later.

Jump ahead

 

Quick setup steps

  1. See the Cloudwatch Metrics document for information about how to configure AWS to feed metrics data.

  2. You must have HTTPS enabled on your web server. AWS allows sending metrics only to HTTPS-enabled web endpoints with a valid certificate (a valid certificate is not expired and is not self-signed).

  3. Copy configuration files to your web server configuration dir (for example, <TOMCAT_DIR>/conf). By default, TNT4J-Streams will try to use the internal configuration dir path /Catalina/localhost/tnt4j-streams:

    1. log4j2.xml - Log4j V2 configuration used by TNT4J-Streams.
    2. tnt4j.properties - Base TNT4J configuration.
    3. tnt4j-common.properties - Common TNT4J configuration.
    4. tnt4j-streams.properties - TNT4J-Streams dedicated TNT4J configuration.
    5. tnt-data-source.xml - TNT4J-Streams data source (stream) configuration. This file has a TNT4J properties configuration section that is dedicated to streams:
      1. The default configuration broadcasts CloudWatch metrics to AutoPilot (sink id ap) and jKool/XRay (sink id jkool) simultaneously. If you want to use just one of these sinks, then change the following <tnt4j-properties>configuration line: 
        <tnt4j-properties>
   <...>
   <!-- FROM -->
   <propertyname="event.sink.factory.BroadcastSequence"value="ap,jkool"/>
   <!-- TO for streaming just to AutoPilot -->
   <propertyname="event.sink.factory.BroadcastSequence"value="ap"/>
   <!-- TO for streaming just to jKool/XRay -->
   <propertyname="event.sink.factory.BroadcastSequence"value="jkool"/>
   <...>
</tnt4j-properties>
      2. Change the AutoPilot sink host value (property event.sink.factory.EventSinkFactory.ap.Host) to match your AutoPilot instance. The default value is localhost.
      3. Change the AutoPilot sink port value (property event.sink.factory.EventSinkFactory.ap.Port) to match your AutoPilot instance. The default value is6060.
      4. Changehttps://data.jkoolcloud.comto your jKool/XRay streaming endpoint URL.
      5. Change the jkool-access-token placeholder to your jKool/XRay streaming token, if you want to stream into that repo.
    6. Parsers.xml - TNT4J-Streams parsers configuration.

    In most cases, you do not need to change the log4j2.xml, tnt4j*.properties, and parsers.xml files. The only file you will need to change is tnt-data-source.xml. 

  1. Download the latest version of the TNT4J-Streams war package from GitHub releases. It will be available in the assets section of the release.

  2. Deploy the tnt4j-streams-<VERSION>.war or tnt4j-streams-servlet-<VERSION>.war file to your web server web-apps dir, for example, <TOMCAT_DIR>/webapps. 

Remove the version token from the war package file name to preserve the web-app context on every deployment.

  1. Start the web server, if it is not already running.

Details: Building the TNT4J-Streams Servlet and Configuring the AWS CloudWatch Metrics Streaming Endpoint

Build TNT4J-Streams servlet

There are two ways to build the TNT4J-Streams war package:

  1. When building the whole tnt4j-streams project, Maven will produce a tnt4j-streams-<VERSION>.war package with all reactor-enabled modules and dependencies built in.
  2. When individually building the tnt4j-streams-servlet module, Maven will produce a tnt4j-streams-servlet-<VERSION>.war package with all module dependencies built in.

Whichever way you choose to build it, the war package contains classes and resources (like web.xml) provided by the tnt4j-streams-servlet module.

Configuration

AWS CloudWatch metrics streaming HTTP endpoint (servlet) configuration is a combination of the following:

  • Web-app deployment descriptor web.xml within a war package (by default) or within your web server configuration dir (for example, <TOMCAT_DIR>/conf/Catalina/localhost/tnt4j-streams).
  • log4j2.xml, which defines the Log4j V2 configuration used by the TNT4J-Streams API.
  • TNT4Jconfiguration properties files split into dedicated scopes (base, common, streams).
  • tnt-data-source.xml, which defines the stream configuration.
  • parsers.xml, which defines parsers used by TNT4-Streams to map metrics fields to the TNT4J activities data model. 

In most cases, you do not need to change the web.xml, log4j2.xml, tnt4j*.properties, and parsers.xml files.

Web-app deployment descriptor (web.xml within war package)

The web-app deployment descriptor defines servlet initialization parameter sets and servlet mapping:

  • init params
    • streams.configs.dir - TNT4J-Streams configuration files location path where tnt4j.properties, log4j2.xml, and tnt-data-source.xml files can be found. The default value is ${catalina.base}/conf/Catalina/localhost/tnt4j-streams.  This setting is optional if you are defining the full path to the individual configuration files with init params tnt4j.config, log4j2.config, and streams.config. 
    • tnt4j.config - TNT4J configuration file path. The default value istnt4j.properties.
    • log4j2.config - Log4j V2 configuration file path. The default value is log4j2.xml
    • streams.config - TNT4J-Streams datasource/parsers configuration file path. The default value is tnt-data-source.xml.
  • servlet mapping
    • The default URL pattern is/.

 

Log4j2

In general, Log4j2 configuration is the same as for common TNT4-Streams logging, except that this config uses the ${sys:catalina.base}/logs dir to locate produced log files.

Also note that loggers are named to match the broadcasting sink ids defined in the stream configuration: ap and jkool instead of prod and qa.

 

TNT4J

In general, TNT4J configuration is the same as for common TNT4-Streams configuration, except that the streams scope (tnt4j-streams.properties file) defines broadcasting sink ids ap and jkool to match the sink target endpoint.

Individual TNT4J streams scope configuration is handled in the <tnt4j-properties> section of the tnt-data-source.xml file.

 

Stream (tnt-data-source.xml)

The major entities in stream configuration are as follows:

  • The CloudWatchMetricsStream entity of class com.jkoolcloud.tnt4j.streams.inputs.HttpServletStreampicks up the HTTP POST transmitted request payload as streamed input data.
  • The ResponseTemplate property defines the stream servlet response template. AWS Kinesis FireHose requires a particular JSON response to ensure successful communication.
  • The tnt4j-propertiessection defines individual stream TNT4J configuration:
    • The event.sink.factory.BroadcastSequence property defines the produced activities broadcasting sinks. The default configuration for this property includes sinks for both AutoPilot (sink id ap) and jKool/XRay (sink id jkool). Therefore the default set of sinks to broadcast stream produced activities is ap,jkool. If you want to use just one of these sinks, then change the following <tnt4j-properties> configuration: 
      <tnt4j-properties>
<...>
<!-- FROM -->
<propertyname="event.sink.factory.BroadcastSequence"value="ap,jkool"/>
<!-- TO for streaming just to AutoPilot -->
<propertyname="event.sink.factory.BroadcastSequence"value="ap"/>
<!-- TO for streaming just to jKool/XRay -->
<propertyname="event.sink.factory.BroadcastSequence"value="jkool"/>
<...>
</tnt4j-properties>
    • The properties group starting with event.sink.factory.EventSinkFactory.ap defines the AutoPilot dedicated sink configuration:
      • Change the AutoPilot sink (id ap) host value (property event.sink.factory.EventSinkFactory.ap.Host) to match your AutoPilot instance. The default value is localhost.
      • Change the AutoPilot sink (id ap) port value (property event.sink.factory.EventSinkFactory.ap.Port) to match your AutoPilot instance. The default value is6060.
    • The properties group starting with event.sink.factory.EventSinkFactory.jkool defines jKool/XRay dedicated sink configuration:
      • Change the jKool/XRay sink (id jkool) URL value (property event.sink.factory.EventSinkFactory.jkool.Url) to match your jKool/XRay instance. The default value ishttps://data.jkoolcloud.com.
      • Change the jKool/XRay sink (id jkool) token placeholder value (property event.sink.factory.EventSinkFactory.jkool.Token) to your jKool/XRay streaming token, if you want to stream into that repo. The placeholder value is jkool-access-token.
  • KinesisFirehoseParser Reference to bootstrap parser of incoming metrics data package.

 

Parsers (parsers.xml)

The parsers unwrap the metrics data package sent by AWS Kinesis FireHose into metrics JSON lines. This allows for metrics to be filtered (see the Metrics filtering section) and for the field values from the metrics lines to be parsed into TNT4J activities and snapshots.

This file defines these parsers:

  • KinesisFirehoseParser - Bootstrap parser that recognizes the format of the received metrics data package.
  • KinesisFirehoseParserStr - Parses Kinesis Firehose JSON metrics batch package where metrics data package is a Base64 encoded string.
  • MetricsParserStr - Filters metric lines before passing them for further parsing.
  • MetricEntryParserAP - Builds metrics data wrapping ACTIVITY entity for AutoPilot sink, containing SNAPSHOTs representing individual metrics lines (one snapshot per metrics line).
  • MetricLineParserAP - Builds metrics line parsed SNAPSHOT entity for AutoPilot sink.
  • MetricEntryParserXRay - Builds metrics data wrapping ACTIVITY entity for jKool/XRay sink, containingSNAPSHOTs representing individual metrics line.
  • MetricLineParserXRay - Builds metrics line parsed SNAPSHOT entity for jKool/XRay sink.
  • ValueParser - Metrics line value fields group parser.
  • DimensionsParserAP - Metrics line dimensions fields group parser for AutoPilot sink.
  • DimensionsParserXRay - Metrics line dimensions fields group parser for jKool/XRay sink.

Metrics filtering

You may want to pick just a set of provided metrics to be streamed. In this case, you can filter metrics. Filtering logic can be done in the  MetricsParserStr parser field (embedded activity) MetricsData transformation. The default filter removes all available empty lines:

boolean pass =StringUtils.isNotEmpty(line);

Additionally, you can define regular expressions (RegEx) to match metrics lines you want to stream. For example. to pick only kafka_server metrics, you would specify the following:

pass &=Matchers.evaluate("regex:kafka_server_.+", line);

Or you can pick some metrics by name for a particular AWS service. For example, if the service is AWS/Kafka, then you can pick a set of metric names like this:

String[] kafkaIncludeMetrics =newString[] {
        "ActiveControllerCount",
        "CpuIdle",
        "CpuSystem",
        "CpuUser",
        "GlobalPartitionCount",
        "GlobalTopicCount",
        "KafkaAppLogsDiskUsed",
        "KafkaDataLogsDiskUsed",
        "MemoryBuffered",
        "MemoryCached",
        "MemoryFree",
        "MemoryUsed",
        "NetworkRxDropped",
        "NetworkRxErrors",
        "NetworkRxPackets",
        "NetworkTxDropped",
        "NetworkTxErrors",
        "NetworkTxPackets",
        "OfflinePartitionsCount",
        "RootDiskUsed",
        "SwapFree",
        "SwapUsed",
        "ZooKeeperRequestLatencyMsMean",
        "ZooKeeperRequestLatencyMsMean"
};

if (StringUtils.contains(line, "AWS/Kafka")) {
   pass &=StringUtils.containsAny(line, kafkaIncludeMetrics);
}

Streamed metrics data

AutoPilot facts

For CloudWatch provided metrics package, TNT4J-Streams produces an  ACTIVITY entity containing the following fields:

  • Name has value AWSCloudWatchMetrics
  • RoutePath has value ap
  • StartTime/EndTime have the current stream runtime timestamp
  • DataCenter has value Amazon_AWS

Metrics lines are placed as child SNAPSHOTs of that ACTIVITY entity. The Snapshot contains the following set of fields:

  • StartTime/EndTime value from metrics line field timestamp
  • Unit value from metrics line field unit
  • EventName provides stream name and set of properties key/value pairs used to lay out AP tree nodes (see property event.sink.factory.EventSinkFactory.ap.Formatter.PathLevelAttributes):
    • MetricStreamName value from metrics line field metric_stream_name as metrics domain
    • namespace=${Namespace}value from metrics line field  namespace
    • region=${Region}value from metrics line field region
    • cluster=${ClusterName}value from dimensions group field Cluster Name
    • broker=${BrokerId}value from dimensions group field Broker ID. If value is numeric then number is prefixed by b-
    • topic=${Topic}value from dimensions group field Topic
    • streamName=${DeliveryStreamName}value from dimensions group field DeliveryStreamName
    • cGroup=${ConsumerGroup}value from dimensions group field Consumer Group
    • cAuth=${ClientAuth}value from dimensions group field Client Authentication
    • metric=${MetricName}value from metrics line field metric_name
  • The Metrics line field group value produces the following fields:
    • Max value from group field max
    • Min value from group field min
    • Sum value from group field sum
    • Count value from group field count
    • P99value from group fieldp99
    • P99_9value from group fieldp99.9
    • Median value from group field TM(25%:75%)

jKool/XRay activities

  • For CloudWatch-provided metrics package, TNT4J-Streams produces an ACTIVITY entity containing the following fields:

    • Name has value AWSCloudWatchMetrics
    • RoutePath has value xray
    • StartTime/EndTime use current stream runtime timestamp
    • DataCenter has value Amazon_AWS

  • Metrics lines are placed as child SNAPSHOTs of that ACTIVITY entity. The Snapshot contains the following set of fields:

    • MetricStreamName value from metrics line field metric_stream_name
    • UserName value from metrics line field account_id
    • Region value from metrics line field region
    • Namespace value from metrics line field namespace
    • MetricName value from metrics line field metric_name
    • StartTime/EndTime value from metrics line field timestamp
    • Unit value from metrics line field unit
    • Category has value AWSCloudWatchMetric
    • The Metrics line field group dimensions produces the following fields:
      • BrokerId value from group field Broker ID. If the value is numeric, then the number is prefixed by b-.
      • InstanceId value from group field InstanceId
      • ClusterName value from group field Cluster Name
      • Topic value from group field Topic
      • ConsumerGroup value from group field Consumer Group
      • ClientAuth value from group field Client Authentication
      • DeliveryStreamName value from group field DeliveryStreamName
    • The Metrics line field group value produces the following fields:
      • Max value from group field max
      • Min value from group field min
      • Sum value from group field sum
      • Count value from group field count
      • P99 value from group field p99
      • P99_9 value from group fieldp99.9
      • Median value from group field TM(25%:75%)
    • Name combines the Namespace, Region ,ClusterName ,BrokerId , and MetricName fields, delimited by|.
    • Guid combines the  StartTime field (formatted as yyyy-MM-dd_HH:mm:ss in the UTC timezone) with the Name field values delimited by|
    • The rest of the metrics line fields are mapped into snapshot properties using their original names 1:1. Usually these fields are either new or very rare.