Recently we had a customer that had issues with a filled disk on the server hosting their Docker pmm-server environment. They were not able to access the web UI, or even stop the pmm-server container because they had filled the /var/ mount point.
Setting correct expectations
The best way to avoid these kinds of issues in the first place is to plan ahead, and to know exactly with what you are dealing with in terms of disk space requirements. Michael Coburn has written a great blogpost on this matter:
We are now using Prometheus version 2 inside PMM server, so you should take it with a pinch of salt. On the other hand, it will show how you should plan ahead, and think about the “steady state” disk usage, so it’s a good read.
That’s the first step to make sure you won’t get into trouble down the line. But, what happens if you are already in trouble? We’ll see two quick ways that may help reclaiming space.
Before anything else, you should stop any and all PMM clients running, so that you don’t have a race condition after recovering some space, in which metrics coming from the running clients will fill up whatever disk you had freed.
If pmm-admin stop --all won’t work, you can stop the services manually, or even manually kill the running processes as a last resort:
1 2 | shell> systemctl list-unit-files | grep enabled | grep pmm | awk '{print $1}' | xargs -n 1 systemctl stop shell> ps ax | egrep "exporter|qan-agent|pmm" | grep -v "ssh" | awk '{print $1}' | xargs kill |
Removing unused containers
In order for the next steps to be as effective as possible, make sure there are no unused containers running, or stopped:
1 | shell> docker ps -a |
If you see any container that you know you don’t need anymore:
1 2 | shell> docker stop <container_name> shell> docker rm -v <container_name> |
WARNING! Do not remove the pmm-data container!
Reclaiming space from unused Docker images
After you are done cleaning unused containers, we can move forward with removing unused images. Unless you are manually building your own Docker images, it’s really easy to get them again if needed, so you shouldn’t be afraid of deleting the ones that are not being used. In fact, you don’t need to explicitly download the images. By simply running docker run … image_name Docker will automatically do it for you if it’s not found locally.
1 2 3 4 5 6 | shell> docker image prune -a WARNING! This will remove all images without at least one container associated to them. Are you sure you want to continue? [y/N] y Deleted Images: ... Total reclaimed space: 3.97GB |
Not too bad, we just reclaimed 4Gb of disk space. This alone should be enough to restart the Docker service and have the pmm-server container back up. But we want more, just because we can 🙂
Reclaiming space from orphaned Docker volumes
By default, when removing a container (with docker rm ) Docker will not delete the associated volumes, unless you use the -v switch as we did above. This will mean that, unless you were aware of this fact, you will probably have some other gigabytes worth of data occupying disk space. We can easily do this with the volume prune command:
1 2 3 4 5 6 | shell> docker volume prune WARNING! This will remove all local volumes not used by at least one container. Are you sure you want to continue? [y/N] y Deleted Volumes: ... Total reclaimed space: 115GB |
Yeah… that’s some significant amount of disk space we just reclaimed back! Again, make sure you don’t care about any of the volumes from your past containers to be able to do this safely, since there is no turning back from this, obviously.
For earlier versions of Docker where this command is not available, you can check this link.
Planning ahead
As mentioned before, you should now revisit Michael’s blogpost, and set the metrics retention and queries retention variables to whatever makes sense for your environment. Even if you plan ahead, you may not be counting on the additional variable overhead of images and orphaned volumes, so you may want to (warning: shameless plug for my own blogpost ahead) use different mount points for your PMM deployment, and avoid using the shared /var/lib/docker/ mount point for it.
PMM also includes a Disk Space usage dashboard, that you can use to monitor this.
Don’t forget to start back up your PMM clients, and continue to monitor them 24×7!
Photo by Andrew Wulf on Unsplash
great post i will share it on Facebook