Starting with Percona Monitoring and Management (PMM) 2.12, we use VictoriaMetrics instead of Prometheus as a metrics store. This blog is written as an FAQ to explain our reasoning for this change as well as to provide answers to various questions on what it may mean for you.
Why is Percona Monitoring and Management moving from Prometheus to VictoriaMetrics?
The main reason to move to VictoriaMetrics is to be able to push metrics from clients to PMM server (see Foiled by the Firewall: A Tale of Transition From Prometheus to VictoriaMetrics), Additionally, VictoriaMetrics offers better performance, lower space utilization, can be deployed in scale-out fashion and has supported MetricsQL – a more expressive variant of PromQL, which is better suited for PMM needs. Note: while VictoriaMetrics may consume slightly more CPU on ingest, it has better query execution performance.
What is the first version of PMM to include VictoriaMetrics instead of Prometheus?
PMM 2.12 is the first version to ship VictoriaMetrics.
Can I continue running Prometheus instead of VictoriaMetrics in PMM 2.12+?
No. There is no option to use Prometheus in PMM v 2.12+. This change is mostly transparent though, so you should not need to.
What do I need to do to migrate my existing data from Prometheus to VictoriaMetrics?
You do not need to do anything, as Percona has implemented transparent migration. VictoriaMetrics will read from your existing Prometheus files and store new data in VictoriaMetrics format so you will not need to do anything manually.
Will Prometheus to VictoriaMetrics conversion require significant downtime for large data sets?
There is no downtime required. Over time, data will be transparently migrated from Prometheus to VictoriaMetrics It is the closest thing to magic you’ve seen. 🙂
Are my older PMM clients compatible with VictoriaMetrics-based PMM?
Yes, existing 2.x clients are supported with PMM 2.12 until further notice. For best compatibility, though, we recommend upgrading your clients too when you update the server.
Will my custom dashboards continue to work now that PMM uses VictoriaMetrics instead of Prometheus?
Yes, 99% of custom dashboards will continue to work. There are some MetricQL differences from PromQL which can impact your dashboards in very rare cases, and it should be easy to fix.
Can I continue using my external exporters with Victoria Metrics-based PMM?
Yes. External exporters continue to be supported.
Will VictoriaMetrics require more resources than Prometheus?
VictoriaMetrics is generally more resource-efficient compared to Prometheus. It will use less CPU, Memory, and consume less disk space.
How do I troubleshoot VictoriaMetrics inside PMM?
There is “VictoriaMetrics” dashboard in the “Insights” category which provides a lot of diagnostic information for VictoriaMetrics. You can also find VictoriaMetrics logs in Server Diagnostics which you can download from the “PMM Settings” page.
I was using the Prometheus built-in web interface for diagnostics, what should I use instead?
There is no exact feature equivalent for this internal Prometheus interface. However, there are ways to get most of the same things done. Check out this document for more information.
Will PMM 2.12 “push” metrics data from client to server?
No. In PMM 2.12 we replaced the time series engine but did not switch to pushing metrics from a client by default; this will happen in future releases. If you want to switch to a push model now you can do it by using pmm-admin setup … –metrics-mode=auto when adding the node. For more information see this document.
What will happen to my custom Prometheus configuration?
VictoriaMetrics supports a lot of Prometheus configuration options worth changing, such as scrape config. If you supplied custom settings in /srv/prometheus/prometheus.base.yml which are not supported by VictoriaMetrics, you need to remove them before upgrading from PMM which included Prometheus to one which uses VictoriaMetrics. Otherwise, VictoriaMetrics process will fail to start.
Can I supply a custom configuration for the VictoriaMetrics engine as I could with Prometheus?
Same as you could supply custom configuration for Prometheus you can supply custom VictoriaMetrics configuration in /srv/prometheus/prometheus.base.yml. For compatibility reasons, the file name was not changed.
Are custom Prometheus recording rules still supported with VictoriaMetrics?
Yes, while technical implementation changed, recording rules will continue to work. See How to Use Prometheus Recording Rules With Percona Monitoring and Management for more details.
Will my alerting rules continue to work?
Yes. Prometheus alerting rules will continue to work using VictoriaMetrics native compatibility with Prometheus alerting functionality. Vmalert is added inside PMM to support this.
Have more questions? Feel free to post them as comments on this blog post or check out our forums.
Not using Percona Monitoring and Management yet?
Hi how do we configure alerts to Microsoft Teams or PagerDuty? till version 2.12 I was able to send alerts on email and teams after I upgraded to 2.13 I dont get any alerts.
these is my current setup for Alertmanager
global:
smtp_smarthost: ‘.protection.outlook.com:25’
smtp_from: ‘[email protected]’
route:
receiver: ‘prometheus-msteams’
group_by: [‘alertname’,’cluster’]
group_wait: 15s
group_interval: 10m
repeat_interval: 1h
routes:
# – match_re:
# alertname: ^(Instance Down|MysqlDown)$
# receiver: ‘prometheus-msteams’
– match_re:
alertname: ^(Instance Down|MysqlDown)$
receiver: ‘opensourcedb-email’
receivers:
– name: opensourcedb-email
email_configs:
– to: ‘[email protected]’
send_resolved: true
– name: ‘pagerduty’
pagerduty_configs:
– service_key:
– name: ‘prometheus-msteams’
webhook_configs: # https://prometheus.io/docs/alerting/configuration/#webhook_config
– send_resolved: true
url: ‘http://1:2000/alertmanager’ # the prometheus-msteams proxy
Can you please clarify how did you supply this configuration in PMM ? Thanks!