Disclaimer: This blog post is about migrating Percona Monitoring and Management 2 (PMM) data between PMM2 versions, and not for migrating data from PMM1 to PMM2. Restoring data from PMM1 to PMM2 is NOT supported since there were many architectural changes.
I recently worked on a customer case where he was not using a pmm-data container mounted in /srv as instructed in the official doc. PMM2 data is stored under /srv (changed from PMM1, which stored data in other directories and had different mounts for pmm-data), which means that if the content of /srv is not mounted on a separate container, upgrade or recreation of pmm-server will lose the existing data.
The procedure of backing up data stored in pmm-server and recreating pmm-server + pmm-data containers with the correct mount in /srv and no data loss is described below, but first, you should check if pmm-data is correctly mounted:
Checking if pmm-data is Holding /srv:
If you are using pmm-data container mounted on “Destination: /srv/”, you should see the following outputs while checking pmm-data:
1 2 3 | [root@centos vagrant]# docker inspect pmm-data | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data", "Destination": "/srv", |
And if pmm-server is using pmm-data, then the volume filename “Source” should match the one from pmm-data:
1 2 3 | [root@centos vagrant]# docker inspect pmm-server | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/6df58f0ad81199277892c2285e870f3edf563cc23ad79277633e26a593171d47/_data", "Destination": "/srv", |
If you are not using pmm-data mounted in /srv, then the output will be showing a different mount (or no mount at all) such as:
1 2 3 | docker inspect pmm-server | egrep "Source|Destination" "Source": "/var/lib/docker/volumes/e0b4cd1c5c65737a24ee10d9e84a6d17a0fc5beff028410c80599840d2c89e67/_data", "Destination": "/var/lib/grafana", |
Backup Existing Data
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # create directory for backing up current PMM2 data mkdir -p /root/pmm-data-backup/srv # stop pmm-server docker stop pmm-server # copy PMM2 data outside the container docker cp pmm-server:/srv /root/pmm-data-backup # rename container as a backup docker rename pmm-server pmm-server-backup # if having a pmm-data container without mount in /srv, then you should backup it up docker rename pmm-data pmm-data-backup |
Restore Into a New pmm-data Mounted in /srv:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | # create pmm-data container mounted in /srv docker create \ -v /srv \ --name pmm-data \ percona/pmm-server:2 /bin/true # restore data into pmm-data docker cp /root/pmm-data-backup/srv pmm-data:/ # start new pmm-container using pmm-data container sudo docker run -d \ -p 443:443 \ --volumes-from pmm-data \ --name pmm-server \ --restart always \ percona/pmm-server:2 # restore permissions docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown root:pmm /srv/clickhouse docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/grafana.db docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown grafana:grafana /srv/grafana/png docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown pmm:pmm /srv/logs docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown postgres:postgres /srv/logs/postgresql.log docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R postgres:postgres /srv/postgres docker run --rm --volumes-from pmm-data -it percona/pmm-server:2 chown -R pmm:pmm /srv/prometheus |
Results should be similar to this:
1 2 3 4 5 6 7 8 9 10 11 | docker exec -it pmm-server ls -l /srv total 12 drwxr-xr-x. 3 root root 47 Jun 9 12:49 alertmanager drwxr-xr-x. 9 root pmm 140 Jun 10 22:37 clickhouse drwxrwxr-x. 3 grafana grafana 69 Jun 10 22:27 grafana drwxrwxr-x. 2 pmm pmm 4096 Jun 10 22:26 logs drwxr-xr-x. 2 root root 115 Jun 9 12:48 nginx -rw-r--r--. 1 root root 6 Jun 9 12:50 pmm-distribution drwx------. 19 postgres postgres 4096 Jun 10 22:37 postgres drwxr-xr-x. 4 pmm pmm 31 Jun 9 12:50 prometheus drwxr-xr-x. 2 root root 6 Jun 9 12:49 update |
Finally, pmm-server needs to be restarted so that it reloads files with correct permissions:
1 2 | # restart pmm-server docker restart pmm-server |
Docker should now be working again with no data loss! And the backup of the containers can be safely removed with:
1 2 | docker rm pmm-server-backup docker rm pmm-data-backup |
Conclusion
It’s strongly suggested to use a separate pmm-data container mounted in /srv for PMM2 to avoid loss of historical data when upgrading/recreating pmm-server. Since the data path for PMM changed in PMM2 compared to PMM1, you might be using a pmm-data container with the wrong mounts (or no pmm-data at all!) which can be easily fixed by using the above procedure.
Hi Carlos,
Can you kindly let me know how to get the O.S metrics like CPU,MEMORY for a cloudsql instance in pmm2