Comments on: Percona Monitoring and Management, Meet Prometheus Alertmanager https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/ Thu, 11 Jun 2020 15:26:19 +0000 hourly 1 https://wordpress.org/?v=6.5.2 By: Steve Hoffman https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10972330 Thu, 11 Jun 2020 15:26:19 +0000 https://www.percona.com/blog/?p=65447#comment-10972330 Elisa, Unfortunately not, we use template variables in our dashboards to make them quite dynamic and the built-in alerting capabilities of grafana does not work with them. So when you create the copy of the dashboard you’re actually stripping out the offending variables and alerting on specific instances.

]]> By: Steve Hoffman https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10972329 Thu, 11 Jun 2020 15:23:40 +0000 https://www.percona.com/blog/?p=65447#comment-10972329 Sai, sorry for the delay…never got an alert that there was a new post until today so let me answer you first! This doesn’t technically resolve template variables…that’s an issue in Grafana but I think I heard that 7.0 lays the foundation to resolve that! What the AlertManager option does is lets you do what you’re after: dynamically alert on systems that meet certain criteria. There’s a ton of great AlertManager recipes online you can find that will allow you to set thresholds and alerting values on any parameters in prometheus you like. CPU over a certain threshold for a certain period of time on a production only system with greater than 12GB of installed RAM…you can alert on that! Want to restrict that alert down to only systems running mongo, no problem. We’re also working on integrating alert manager into PMM as well so you don’t have to have your own setup but not quite ready for production.

]]> By: elisetta1984 https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10972328 Thu, 11 Jun 2020 13:32:54 +0000 https://www.percona.com/blog/?p=65447#comment-10972328 Hello Steve.
Thanks for your explanations. Actually I can add an alert for a new Dashboard.
But is there also the way to add directly an alert for an existing Dashboard without creating a new one?
Thanks and Regards,
Elisa

]]> By: Sai https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10971935 Mon, 16 Mar 2020 16:41:34 +0000 https://www.percona.com/blog/?p=65447#comment-10971935 Very nice article.

Couple of questions —

1. Does this resolve template variables issue which we have been facing till graphana 4.x?
2. If we have 250 mongo servers, How can we configure and send alerts only to those servers where there is issue?

]]> By: Steve Hoffman https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10971880 Tue, 03 Mar 2020 22:14:05 +0000 https://www.percona.com/blog/?p=65447#comment-10971880 elisetta1984,
Grafana Alerting still works in PMM 2.3.0, I was just able to set one up on my test instance and it’s alerting me as expected. I’ll assume you have configured a notification channel, once that’s done, you can create a new dashboard, add a panel to it and the option for alerts isn’t a tab anymore but a “bell” icon along the left side of the “edit panel” dialog boxes (there will be an icon for: Queries, Visualization, General, and finally Alert) . If you’re looking to add alerts to the existing graphs that’s possible but a little more involved as many of our dashboards are templated. Here’s a blog post from PMM1 but the bulk of what’s outlined still holds true in PMM2 https://www.percona.com/blog/2017/02/02/pmm-alerting-with-grafana-working-with-templated-dashboards/

We are also looking at incorporating the latest Grafana 6.6 as there’s been some significant work done with alerting but that’s going to be a few releases out for us.

Hope this helps!
Steve

]]> By: elisetta1984 https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10971873 Mon, 02 Mar 2020 12:55:15 +0000 https://www.percona.com/blog/?p=65447#comment-10971873 Hello.
If I don’t want to use the new Prometheus Alertmanager, is it still possible to use the Grafana Alerting feature? I cannot find anymore the Alert Tab on the dashboard graph panel for PMM 2.3.0.
Thanks and Regards, Elisa

]]> By: Vadim Yalovets https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10971860 Tue, 25 Feb 2020 14:10:03 +0000 https://www.percona.com/blog/?p=65447#comment-10971860 Could you check your expr part directly in prometheus UI?
https:///prometheus/

]]> By: Eugene https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10971859 Tue, 25 Feb 2020 12:28:37 +0000 https://www.percona.com/blog/?p=65447#comment-10971859 ttext in the comment above did not preserve yaml indentation which was correct originally

]]> By: Eugene https://www.percona.com/blog/percona-monitoring-and-management-meet-prometheus-alertmanger/#comment-10971858 Tue, 25 Feb 2020 12:24:51 +0000 https://www.percona.com/blog/?p=65447#comment-10971858 I’m trying to add the following in the Alertmanager rule section:

– alert: MongodbReplicationLag
expr: (avg by (cluster,environment,set)(mongodb_mongod_replset_member_optime_date{state=”PRIMARY”}) – min by (cluster,environment,set) (mongodb_mongod_replset_member_optime_date{state=”SECONDARY”})) > 120
for: 5m
labels:
severity: warning
source: pmmprod

Once I click “Apply Alertmanager settings button” button, the error pops-up: Invalid Alert Manager rules.

What am I doing wrong here?

]]>