Percona Monitoring and Management (PMM2) Server runs as a Docker container, a Virtual appliance, or as an instance on Amazon or Azure cloud services. Here I’ll show how to move the PMM Server and its data from one type to another.
Note, this is only for PMM2 to PMM2—you can’t migrate data from PMM Server version 1 to version 2 because of significant architectural differences.
For this exercise, imagine that your PMM server:
- Is running on an Amazon EC2 instance (Server A) from an AMI,
- You want to move it to a dedicated server (Server B) running as a Docker container.
- Server A monitors one client instance (node1) with a MongoDB service (mongodb1).
Here’s the output of pmm-admin status for this instance.
Export Data
PMM2 data is stored in the /srv folder for all types of installations. So first make a backup archive of it.
1 | tar -cv /srv | gzip > pmm-data.tar.gz |
Copy this archive to Server B.
1 | scp pmm-data.tar.gz user1@172.17.0.2:~/ |
Prepare New Server
Connect to Server B run all further commands on this server. Prepare the Docker container.
1 | docker create -v /srv --name pmm-data percona/pmm-server:2 /bin/true |
Next extract exported data from the archive.
1 | tar -zxvf pmm-data.tar.gz -C /tmp |
Create a container for the new PMM Server with a /srv partition on a separate container ( pmm-data).
1 | docker run -d -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2 |
Stop all services and copy the exported data into the container.
1 2 3 | docker exec -it pmm-server supervisorctl stop all docker exec -it pmm-server sh -c 'cd /; rm -rf /srv/victoriametrics/data' docker cp /tmp/srv pmm-data:/ |
Restore permissions for migrated data folders.
1 2 3 4 | docker exec -it pmm-server chown -R root:pmm /srv/clickhouse /srv/ia /srv/nginx /srv/pmm-distribution /srv/update docker exec -it pmm-server chown -R pmm:pmm /srv/logs /srv/victoriametrics /srv/alertmanager /srv/prometheus docker exec -it pmm-server chown -R grafana:grafana /srv/grafana docker exec -it pmm-server chown -R postgres:postgres /srv/postgres /srv/logs/postgresql.log |
Restart PMM Server so that it reloads files with the correct permissions.
1 | docker restart pmm-server |
Switch Services to New Server
That’s it! Now you can switch your monitored node1 to use the new server (Server B).
Edit the the PMM agent configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml .
Set the IP address of Server B ( 172.17.0.2 ) and restart pmm-agent.
1 | systemctl restart pmm-agent |
Check Status
Check the status of pmm-agent and monitored services with pmm-admin status.
The agent is now connected to your new server.
In the Grafana UI, you can see the migrated data of Server B. (The time gap in the data is how long it took to run the import and switch node1 to the new server.)
If historical data is here then we are done. Otherwise, please follow the commands that are provided in the next section.
Export/Import VictoriaMetrics Data
Copy the metrics in the VictoriaMetrics time-series database using an API request for export/import data. (You can do the export remotely and run all further commands on Server B.)
1 | curl -k -G -u admin:admin https://3.86.222.201/prometheus/api/v1/export/native -d 'match={__name__!=""}' > exported_data.dump |
Next import the VictoriaMetrics data.
1 | curl -k -u admin:admin -X POST https://172.17.0.2/prometheus/api/v1/import/native -T exported_data.dump |
By default, the maximum allowed size of the client request body for PMM Server’s Nginx service is 10Mb. If exported_data.dump is bigger than this you must increase the limit and repeat the import.
1 2 | docker exec -it pmm-server bash -c "sed -i 's/client_max_body_size 10m;/client_max_body_size 1000m;/g' /etc/nginx/conf.d/pmm.conf" docker exec -it pmm-server bash -c "supervisorctl restart nginx" |
Conclusion
You can use the same process to move from any instance type to another. Also, we have got a separate blog post about how to migrate if the pmm-data container isn’t used. Check it out!
