Using Amazon CloudWatch Metric Streams and Amazon Kinesis Data Firehose, you can send CloudWatch metrics into an HTTP End point with a latency of 3 minutes or less.
- Create these AWS resources in your AWS account in the region for which you want to stream CloudWatch metrics:
- Create a Kinesis Data Firehose delivery stream that delivers metrics to an HTTP endpoint, along with destination error logs to Cloudwatch, and backup logs to an S3 account in case delivery fails for any metrics.
- Create a CloudWatch Metric Stream linked to the Firehose delivery stream.
- (Optional.) Specify a limited set of namespaces to stream metrics like S3, Kafka, EC2, and so on.
- Once you create these resources, the destination HTTP endpoint immediately starts receiving the streamed metrics.
Create a new Kinesis Data Firehose delivery stream
Choose your source and destination.
For source, select “Direct PUT or other sources.”
For destination, select “HTTP Endpoint.”
Enter a Delivery stream name. An example is provided below.
(Optional.) Configure Kinesis Data Firehose to transform your record data.
The default setting is Disabled.
Specify the destination settings for your delivery stream.
- (Optional.) Specify the HTTP endpoint name.
- Provide your HTTP endpoint URL.
- (Optional.) Provide the Access key. (Contact the endpoint owner.)
- Content encoding. As explained on the page, Kinesis Data Firehose uses the content encoding to compress the body of a request before sending the request to the destination. (By default, this is Disabled.)
- Retry duration. This is the time period during which Kinesis Data Firehose retries sending data to the selected HTTP endpoint.
- (Optional.) Parameters. Kinesis Data Firehose includes these key-value pairs in each HTTP call.
- Buffer hints. Kinesis Data Firehose buffers incoming records before delivering them to your HTTP endpoint domain, which accepts only the HTTPS protocol. Record delivery is triggered once the value of either of the specified buffering hints is reached.
Specify Backup settings to ensure that the data can be recovered if record processing transformation does not produce the desired results.
Provide Advanced settings, including server-side encryption, Amazon CloudWatch error logging, IAM Role permissions, and Tags.
Click Create delivery stream.
Create your CloudWatch Metric Stream
Choose whether you want to stream all CloudWatch metrics, or choose specific namespaces with “Include” or “Exclude” lists. For example, selected specific namespaces like EC2, Kafka, and S3.
- Select the Firehose you created to use for sending the metrics to the HTTP endpoint.
Create a new service role to put records in Kinesis Data Firehose.
Change the output format to be JSON.
Add additional statistics to include the AWS percentile metrics you would like to send to the Destination.
Enter a name for your metric stream.
Click Create metric stream.
Once you see that the Metric Stream resource has been successfully created, wait five minutes to allow streamed metrics to reach the Destination HTTP endpoint.
Example: Create an Amazon MSK cluster using the AWS Management Console
Sign in to the AWS Management Console, then open the Amazon MSK Console and go to Create cluster.
For the Cluster name, enter "demo-cluster-1".
From the table under General cluster settings, choose the following settings:
- Cluster Type
- Apache Version
- Broker Type
From the table under All cluster settings, copy the values of the following settings and save them. You will need them later in this tutorial:
- Security groups associated with VPC
Click Create cluster.
Check the cluster Status on the Cluster summary page. The status changes from Creating to Active as Amazon MSK provisions the cluster. When the status is Active, you can connect to the cluster.