This blog post contains step-by-step instructions for setting up alerting in Percona Monitoring and Management (PMM) using Grafana.

As a Solutions Engineer at Percona, I get to speak with a lot of the early adopters of Percona’s new PMM software. One of the frequent feature requests from them was the need for alerts at certain MySQL and MongoDB thresholds. Percona’s philosophy on alerting is that we don’t want to reinvent the wheel. We integrated Consul, Grafana and Prometheus into PMM because they are fantastic, free and open-source products. PMM packages them together in an easy to deploy and maintain format.

With the release of Grafana 4.0 came Alerting! The Grafana team excitedly shared this feature with its users (including Percona). Percona quickly followed suit by releasing a new PMM 1.0.7 package that contained Grafana 4.0.

Configuring Grafana for Alerting

The instructions below assume that you have a running instance of PMM. Percona’s website contains detailed installation instructions if you don’t.

There are several ways to receive alerts with Grafana – email, Slack, Webhook and PagerDuty. For this blog post, I demonstrate alerting through email.Please note that for this to work correctly, SMTP settings must be configured in the Grafana config file, which is located inside your docker container. To enter the Docker container, use the ID of the running container found with the command “docker ps”. Once you have the container ID, enter the Docker shell by using the command docker exec -it CONTAINER_ID /bin/bash. From this terminal, you can access the /etc/grafana/grafana.ini configuration file. Documentation for the configuration file can be found on Grafana’s website.

Below is a basic, default configuration setting for the SMTP/email section for testing purposes that works for Gmail:

There are also instructions for setting up other alerting methods on Grafana’s website. Once you have made changes to the config file, you can restart pmm-server to pick up the changes with a docker restart pmm-server command.

Once you restart the docker container, access the Alert Notification panel by selecting the Grafana icon dropdown in the upper left corner of any Grafana dashboard:

Alerting Tab

To set up alerting, you need to create an alert notification on this page. For this test, I am only entering one email address. You can enter more than one by separating emails with commas.

Testing the Configuration

You can test your server’s configuration by clicking the “Send Test” button.

Alert Notification Page

You’ll know that you configured this correctly when you receive the following email:

Test Email Confirmation

Enabling Alerting for a Variable

Now that you’ve updated your Grafana config file and tested to ensure your SMTP settings are correct, it’s time to set up an alert for a specific variable. To do that, go the dashboard for which you’re interested in setting up alerting:

  1. Click the title [1]
  2. Select “Edit” [2]
  3. Choose the Alert tab [3]
  4. Select the “Create Alert” button. In the Alert Config section, you can modify the frequency at which the alert checks for a trigger [4] and the conditions to trigger the alert [5].

Configuring Alert

Currently, a known limitation exists with Grafana. Alerting doesn’t support template variables yet. However, there is a feature request on Github to add this functionality in the future. Because of this, template variables must be removed from the Metrics tab [6] of the dashboard you are editing.

Grafana sends an alert email whenever the configured threshold gets passed:

Alert Email

Once an alert clears, you will get an all clear email.

All Clear Email

Summary

Proactive monitoring is vital to a highly-available environment. Hopefully, this blog post gives you the confidence to move forward with integrating alerting into your PMM platform. I also wrote this blog post to highlight the fact that our PMM team continues to add functionality requested by users with regards to monitoring and maintaining their environment. If you’d like to ask questions or give feedback, the PMM forums on Percona’s website are a good place to go.

4 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
ajeet

As you said, template variable needs to be removed from metric tab. Which on is the template variable in this example? Can you show the variable to be removed from metric tab?

0
Barrett Chambers

Hi ajeet,

Our CEO, Peter Zaitsev, wrote a blog dedicated to working around the template variable limitation that currently exists within Grafana as a follow-up to this post. See his post here: https://www.percona.com/blog/2017/02/02/pmm-alerting-with-grafana-working-with-templated-dashboards/

He gives an example of setting up a fresh dashboard to work around this limitation.

0
utdrmac

You say I need to edit /etc/grafana/grafana.ini yet that file exists INSIDE the docker container. Can you update your instructions for editing this file within the container for those of us that don’t have docker knowledge beyond following the PMM install instructions?

0
Muhammad Rashid

After adding gmail smtp configuration in /etc/grafana/grafana.ini file in container. Once I restart the container then new container id is generated and smtp configuration information is reset. What to do

0