Grafana Alerting in a Multi-cloud World

• It does not allow the administrator to apply an alert against a template variable.
• There is no way to qualify how those alerts should be applied within a new environment if the clouds are substantially different in terms of size, load or traffic.
• When tuning the alerts, there is no simple way to synchronize the changes across the clouds and avoid overriding existing parameters for the alert.

Identifying a Flexible but Simple Solution

• Set thresholds low enough to cover all environments
• Focus on alerts that pertain to failures or latency above some threshold instead of success below a particular threshold
• Use a ratio of failures to successes and alert on a percentage basis (e.g., > n% of API requests failing)
• In the case of Kafka alerts, use the KafkaBacklogAlert (see below) rather than alerting on a specific numeric lag value
• When applicable, emit metrics based on availability rather than activity — e.g., insufficient available resources in a pool, or insufficient available memory. This is because pool sizes, memory and other resources may be configured differently depending on the cloud. As such, it is easier to alert when the availability approaches zero, as opposed to a configurable maximum.

A Closer Look at “Alerts as Code”

Generating Alerts

• Read the common values and the microservice JSON file

   

• Generate dashboards in the Grafana JSON format

   

• Upload dashboards to Grafana

Scraping Alerts

• Automates the process of exporting a Grafana dashboard as JSON
• Finds any alerts specified on the dashboard

   

• Migrates the alerts into our in-house defined alert types

Alert Files

{
    "name": "microservice-name",
    "scope": "pool",
    "notificationChannels": <"slack_quality", "pagerduty_cloud">,
    "includes": <>,
    "excludes": <"cloud2", "cloud3.pool2">,
    "alerts": <
        {
            "alertType": "ThresholdAlert",
            "title": "a series of unfortunate events has occurred!",
            "query": "sumSeries(microservice-name.*.meter.events.unfortunate)",
            "type": "gt",
            "params": <1000>,
            "message": "Olaf is a cruel, scheming man. It is recommended that you escape to find shelter with your quirky Uncle Monty. Or restart microservice-name. Up to you."
        },
        {
            "AlertType": "KafkaBacklogAlert",
            "Title": "Backlog Alert",
            "Message": "service is lagging or the kafka consumer is stopped. This will delay or prevent any dependent processing.",
            "ConsumerGroup": "microservice-consumer-group",
            "EpsMetric": "sumSeries(microservice-name.*.events.processed)",
            "BacklogThreshold": 600
        }
    >
}

Types of Alerts

• ThresholdAlert.

   

  The simplest type of alert, the ThresholdAlert, takes a single query and a threshold. It supports less than, greater than, within range, outside range and no value.
• RawAlert.

   

  For cases where more flexibility is required, the RawAlert contains a set of conditions and targets in Grafana JSON form. This alert is highly verbose, so it should only be used when absolutely necessary.
• KafkaBacklogAlert.

   

  An alert that can be deployed across clouds with differing levels of traffic easily, the KafkaBacklogAlert divides the current lag by the current events/messages per second to understand how much backlog has accumulated.
• KafkaLagAlert. An early Kafka alert type that uses a hard-coded threshold for lag. Mostly useful in scraping existing alerts, this alert has been largely discontinued in favor of KafkaBacklogAlert for increased flexibility.

The Benefits of “Alerts as Code”

• Parity. Our solution allows the team to ensure parity across all clouds, even as alerts are modified or augmented. Alerts can now be updated easily in all places at once, and version control provides an opportunity for formal reviews and viewing change history.
• Simplicity and efficiency. Developing our own JSON allowed our team to define alerts in a simplified form. Our team can also take some shortcuts that make it easier to set up an alert, as compared to doing so manually in Grafana. Likewise, if we need to make changes en masse and enforce standards, we can easily make updates with the tooling rather than doing so manually in Grafana.

Limitations/Challenges

Conclusion