One of the most powerful observability features is dynamic sensors. These allow you to create sensors that automatically react to changes.
Concepts
As an analogy, imagine you are a sheep farmer and you want to prepare your sheep for shearing.
You might want to sort you sheep by size and breed. You have a pen and you only let those that match your criteria into it. Once in the pen, you may also classify further by ease of shearing or other factors.
You can think of a dynamic sensor in the same way, it is a pen that only allows the data you want into it for evaluation.
Subscription
The first step in creating a subscription to the set of data that is to be evaluated. For example, all NYSE stocks, employees that work in the London office, Kafka topics in the payroll application, all sheep in the south field, and so on. You do this in the sensor wizard on the Select Facts To Monitor tab. For the example below, we have a set of climate metrics for cities. By clicking one of the city temperatures, it prefills that into the Fact field.
To configure as a dynamic sensor, we need to switch some of the the static hierarchy to generic references. For example, we could select all cities, US cities, or Texas cities. To select all US, we change TX and Houston to asterisks. In most cases, it is best to deselect Auto Parameterize and then click Include.
Once you do this, we have accomplished which cities we are going to consider, but we have not applied any criteria yet.
That is done on the next tab, click Next to proceed.
Selection
The Create Dynamic Sensors tab defines the criteria on which of the data we subscribed to becomes sensors. In the original analogy, this is the gate into the pen. Of all of the data we collected, which ones are candidates for sensors. To do this, we specify the criteria we want to exist before we create the sensor. For our example, we can pick a temperature such as 60 degrees (Fahrenheit) as the lowest temperature we will consider.
As in the original analogy, we have now identified which of the sheep get into the pen, but have not sorted them based on other criteria. We do that on the next tab, click Next to proceed.
Evaluation
On the Dynamic Sensor Options, you further classify the conditions related to the data. Since we are using a weather example, we will create severity levels based on the temperature.
In this example, there are 4 severities
- Critical - greater than 90
- Error - greater than 80
- Warning - 60 and above
- Success - anything else
Note the following items (which are explained in more detail in Dynamic Sensor Advanced Topics).
- The severities are evaluated in order and the first one matched is used
- The level for Warning matches the criteria for selection for the sensor
- The Success level exists to handle cases where the data changes after the sensor was created
Execution
We would now deploy the policy with the sensor to a policy manager and see the result.
Of course, temperature is not the only metric needed to predict the weather. Dynamic Sensor Using Multiple Metrics covers how to add multiple metrics for dynamic sensors.