Over the course of the years, we see the growing interest in running databases and stateful workloads in Kubernetes. With Container Storage Interfaces (CSI) maturing and more and more Operators appearing, running stateful workloads in your favorite platform is not that scary anymore. Our Kubernetes story at Percona started with Operators for MySQL and MongoDB, adding PostgreSQL later on.
Percona Monitoring and Management (PMM) is an open source database monitoring, observability, and management tool. It can be deployed in a virtual appliance or a Docker container. Our customers requested us to provide a way to deploy PMM in Kubernetes for a long time. We had an unofficial helm chart which was created as a PoC by Percona teams and the community (GitHub).
We are introducing the Technical Preview of the helm chart that is supported by Percona to easily deploy PMM in Kubernetes. You can find it in our helm chart repository here.
Use cases
Single platform
If Kubernetes is a platform of your choice, currently you need a separate virtual machine to run Percona Monitoring and Management. No more with an introduction of a helm chart.
As you know, Percona Operators all have integration with the PMM which enables monitoring for databases deployed on Kubernetes. Operators configure and deploy pmm-client sidecar container and register the nodes on a PMM server. Bringing PMM into Kubernetes simplifies this integration and the whole flow. Now the network traffic between pmm-client and PMM server does not leave the Kubernetes cluster at all.
All you have to do is to set the correct endpoint in a pmm section in the Custom Resource manifest. For example, for Percona Operator for MongoDB, the pmm section will look like this:
1 2 3 4 | pmm: enabled: true image: percona/pmm-client:2.28.0 serverHost: monitoring-service |
Where monitoring-service is the service created by a helm chart to expose a PMM server.
High availability
Percona Monitoring and Management has lots of moving parts: Victoria Metrics to store time-series data, ClickHouse for query analytics functionality, and PostgreSQL to keep PMM configuration. Right now all these components are a part of a single container or virtual machine, with Grafana as a frontend. To provide a zero-downtime deployment in any environment, we need to decouple these components. It is going to substantially complicate the installation and management of PMM.
What we offer instead right now are ways to automatically recover PMM in case of failure within minutes (for example leveraging the EC2 self-healing mechanism).
Kubernetes is a control plane for container orchestration that automates manual tasks for engineers. When you run software in Kubernetes it is best if you rely on its primitives to handle all the heavy lifting. This is what PMM looks like in Kubernetes:
- StatefulSet controls the Pod creation
- There is a single Pod with a single container with all the components in it
- This Pod is exposed through a Service that is utilized by PMM Users and pmm-clients
- All the data is stored on a persistent volume
- ConfigMap has various environment variable settings that can help to fine-tune PMM
In case of a node or a Pod failure, the StatefulSet is going to recover PMM Pod automatically and remount the Persistent Volume to it. The recovery time depends on the load of the cluster and node availability, but in normal operating environments, PMM Pod is up and running again within a minute.
Deploy
Let’s see how PMM can be deployed in Kubernetes.
Add the helm chart:
1 2 | helm repo add percona https://percona.github.io/percona-helm-charts/ helm repo update |
Install PMM:
1 | helm install pmm --set service.type="LoadBalancer" percona/pmm |
You can now login into PMM using the LoadBalancer IP address and use a randomly generated password stored in a pmm-secret Secret object (default user is admin).
The Service object created for PMM is called monitoring-service:
1 2 3 4 5 6 7 8 9 10 11 12 | $ kubectl get services monitoring-service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE monitoring-service LoadBalancer 10.68.29.40 108.59.80.108 443:32591/TCP 3m34s $ kubectl get secrets pmm-secret -o yaml apiVersion: v1 data: PMM_ADMIN_PASSWORD: LE5lSTx3IytrUWBmWEhFTQ== … $ echo 'LE5lSTx3IytrUWBmWEhFTQ==' | base64 --decode && echo ,NeI<w#+kQ`fXHEM |
Login to PMM by connecting to HTTPS://<YOUR_PUBLIC_IP>.
Customization
Helm chart is a template engine for YAML manifests and it allows users to customize the deployment. You can see various parameters that you can set to fine-tune your PMM installation in our README.
For example, to set choose another storage class and set the desired storage size, set the following flags:
1 2 3 | helm install pmm percona/pmm --set storage.storageClassName="premium-rwo" -–set storage.size=”20Gi” |
You can also change these parameters in values.yaml and use “-f” flag:
1 2 3 4 5 6 7 | # values.yaml contents storage: storageClassName: “premium-rwo” size: 20Gi <span style="font-weight: 400;">helm install pmm percona/pmm -f values.yaml</span> |
Maintenance
For most of the maintenance tasks, regular Kubernetes techniques would apply. Let’s review a couple of examples.
Compute scaling
It is possible to vertically scale PMM by adding or removing resources through pmmResources variable in values.yaml.
1 2 3 4 5 6 7 | pmmResources: requests: memory: "4Gi" cpu: "2" limits: memory: "8Gi" cpu: "4" |
Once done, upgrade the deployment:
1 | helm upgrade -f values.yaml pmm percona/pmm |
This will restart a PMM Pod, so better plan it carefully not to disrupt your team’s work.
Storage scaling
This depends a lot on your storage interface capabilities and the underlying implementation. In most clouds, Persistent Volumes can be expanded. You can check if your storage class supports it by describing it:
1 2 3 | kubectl describe storageclass standard … AllowVolumeExpansion: True |
Unfortunately, just changing the size of the storage in values.yaml (storage.size) will not do the trick and you will see the following error:
1 2 | helm upgrade -f values.yaml pmm percona/pmm Error: UPGRADE FAILED: cannot patch "pmm" with kind StatefulSet: StatefulSet.apps "pmm" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden |
We use the StatefulSet object to deploy PMM, and StatefulSets are mostly immutable and there are a handful of things that can be changed on the fly. There is a trick though.
First, delete the StatefulSet, but keep the Pods and PVCs:
1 | kubectl delete sts pmm --cascade=orphan |
Recreate it again with the new storage configuration:
1 | helm upgrade -f values.yaml pmm percona/pmm |
It will recreate the StatefulSet with the new storage size configuration.
Edit Persistent Volume Claim manually and change the storage size (the name of the PVC can be different for you). You need to change the storage in spec.resources.requests.storage section:
1 | kubectl edit pvc pmm-storage-pmm-0 |
The PVC is not resized yet and you can see the following message when you describe it:
1 2 3 4 5 6 | kubectl describe pvc pmm-storage-pmm-0 … Conditions: Type Status LastProbeTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- FileSystemResizePending True Mon, 01 Jan 0001 00:00:00 +0000 Thu, 16 Jun 2022 11:55:56 +0300 Waiting for user to (re-)start a pod to finish file system resize of volume on node. |
The last step would be to restart the Pod:
1 | kubectl delete pod pmm-0 |
Upgrade
Running helm upgrade is a recommended way. Either once a new helm chart is released or when you want to upgrade the newer version of PMM by replacing the image in the image section.
Backup and restore
PMM stores all the data on a Persistent Volume. As said before, regular Kubernetes techniques can be applied here to backup and restore the data. There are numerous options:
- Volume Snapshots – check if it is supported by your CSI and storage implementation
- Third-party tools, like Velero, can handle the backups and restores of PVCs
- Snapshots provided by your storage (ex. AWS EBS Snapshots) with manual mapping to PVC during restoration
What is coming
To keep you excited there are numerous things that we are working on or have planned to enable further Kubernetes integrations.
OpenShift support
We are working on building a rootless container so that OpenShift users can run Percona Monitoring and Management there without having to grant elevated privileges.
Microservices architecture
This is something that we have been discussing internally for some time now. As mentioned earlier, there are lots of components in PMM. To enable proper horizontal scaling, we need to break down our monolith container and run these components as separate microservices.
Managing all these separate containers and Pods (if we talk about Kubernetes), would require coming up with separate maintenance strategies. This brings up the question of creating a separate Operator for PMM only to manage all this, but it is a significant effort. If you have an opinion here – please let us know on our community forum.
Automated k8s registration in DBaaS
As you know, Percona Monitoring and Management comes with a technical preview of Database as a Service (DBaaS). Right now when you install PMM on a Kubernetes cluster, you still need to register the cluster to deploy databases. We want to automate this process so that after the installation you can start deploying and managing databases right away.
Conclusion
Percona Monitoring and Management enables database administrators and site reliability engineers to pinpoint issues in their open source database environments, whether it is a quick look through the dashboards or a detailed analysis with Query Analytics. Percona’s support for PMM on Kubernetes is a response to the needs of our customers who are transforming their infrastructure.
Some useful links that would help you to deploy PMM on Kubernetes:
- Helm chart Github repository
- Screencast demoing the deployment
- Community Forum for open discussions
- JIRA to raise a bug or improvement suggestion
Excellent youtube video. The example to use the service type was just perfect. This wasnt documented elsewhere and some k8s platforms need loadbalancer service instead of nodeport. Thanks for video and documentation.