In Percona Monitoring and Management 2.12.0 (PMM), we replaced Prometheus with VictoriaMetrics. Metrics data can now be ‘pushed’ to the server as well as being ‘pulled’ by the server.
This is useful if you want to keep open ports to a minimum or your monitored system is in an environment that favors “inside–>out” communication (like Kubernetes).
When adding new services for monitoring, you can use the --metrics-mode=push flag of pmm-admin and starting with v 2.14.0 push will be the default mode if you don’t specify. (Note: Installing the 2.14.0 or newer PMM server will change the default behavior on 2.12.0 and 2.13.0 clients from “pull” method to “push” for any newly added services, existing services will remain whatever mode they were in prior to upgrading.)
But what if you need to change the metrics mode for an existing service? You must first remove it then re-add it with the same name and some extra flags, and I wrote this blog to show how easy that process is.
I’ll explain using my test installation that has three monitored services already added in pull mode.
1 | pmm-admin list |
Carefully make a note of the service names to be switched to push mode. (I write carefully because if the service name is accidentally changed, the system will treat this service as a new one and both new and old services will show up on your dashboards.)
Here’s the switch for the mysql_8_dev service. (I’ve highlighted the metrics-mode option.)
1 2 3 | pmm-admin remove mysql mysql_8_dev; pmm-admin add mysql --username="root" --password="ps" --host="172.17.0.3" \ --metrics-mode=push mysql_8_dev |
You can use similar commands for PostgreSQL services:
1 2 | pmm-admin remove postgresql PSQL-Stg; pmm-admin add postgresql --port="5432" --metrics-mode=push PSQL-Stg |
Now, check that these two changed services are working in the new mode:
Node exporters can also be changed to work in push mode.
However, this involves all services on the node, due to the node_id changing. So, you need an extra step: collect the details of currently monitored services on the node, including all the authentication credentials your services may use for exporters’ access.
When you have your list of services, with their names, sockets, and credentials (which you will need after this step) you can run the re-configuration command for the node:
1 2 3 4 | pmm-agent setup --config-file="/usr/local/percona/pmm2/config/pmm-agent.yaml" \ --server-insecure-tls --server-username=admin --server-password=admin \ --server-address=172.17.0.2:443 --metrics-mode=push 172.17.0.1 generic Dell-5480 \ --force |
(Note the use of the --force flag, used for removing the node with this name and creating a new one.)
Now, the node is in push mode and you must re-add services using the same names as were used previously. (The commands are the same as those shown above for adding services.)
When all services have been added, check the current state of modes:
Here you see all exporters are working in push mode.
You’ll see the new metrics data for your services on dashboards after a short wait:
I hope that shows how easy it is to switch from pull to push modes, so long as you use the same names as before.
