There are two Operators at Percona to deploy MySQL on Kubernetes:
- one is based on Percona XtraDB Cluster (PXC), production-ready and widely adopted
- another is based on Percona Server for MySQL, now in tech preview and not recommended for production yet
We wrote a blog post in the past explaining the thought process and reasoning behind creating the new Operator for MySQL. The goal for us is to provide production-grade solutions to run MySQL in Kubernetes and support various replication configurations:
- Synchronous replication
- with Percona XtraDB Cluster
- with Group Replication
- Asynchronous replication
With the latest 0.2.0 release of Percona Operator for MySQL (based on Percona Server for MySQL), we have added Group Replication support. In this blog post, we will briefly review the design of our implementation and see how to set it up.
Design
This is a high-level design of running MySQL cluster with Group Replication:
MySQL Router acts as an entry point for all requests and routes the traffic to the nodes.
This is a deeper look at how the Operator deploys these components in Kubernetes:
Going from right to left:
- StatefulSet to deploy a cluster of MySQL nodes with Group Replication configured. Each node has its storage attached to it.
- Deployment object for stateless MySQL Router.
- Deployment is exposed with a Service. We use various TCP ports here:
- MySQL Protocol ports
- 6446 – read/write, routing traffic to Primary node
- 6447 – read-only, load-balancing the traffic across Replicas
- MySQL X Protocol – can be useful for CRUD operations, ex. asynchronous calls. Ports follow the same logic:
- 6448 – read/write
- 6449 – read-only
- MySQL Protocol ports
Action
Prerequisites: you need a Kubernetes cluster. Minikube would do.
The files used in this blog post can be found in this Github repo.
Deploy the Operator
1 | kubectl apply --server-side -f https://raw.githubusercontent.com/spron-in/blog-data/master/ps-operator-gr-demo/bundle.yaml |
Note –server-side flag, without it you will get the error:
1 | The CustomResourceDefinition "perconaservermysqls.ps.percona.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes |
Our Operator follows OpenAPIv3 schema to have proper validation. This unfortunately increases the size of our Custom Resource Definition manifest and as a result, requires us to use –server-side flag.
Deploy the Cluster
We are ready to deploy the cluster now:
1 | kubectl apply -f https://raw.githubusercontent.com/spron-in/blog-data/master/ps-operator-gr-demo/cr.yaml |
I created this Custom Resource manifest specifically for this demo. Important to note variables:
- Line 10: clusterType: group-replication – instructs Operator that this is going to be a cluster with Group Replication.
- Lines 31-47: are all about MySQL Router. Once Group Replication is enabled, the Operator will automatically deploy the router.
Get the status
The best way to see if the cluster is ready is to check the Custom Resource state:
1 2 3 | $ kubectl get ps NAME REPLICATION ENDPOINT STATE AGE my-cluster group-replication 35.223.42.238 ready 18m |
As you can see, it is ready. You can also see initializing if the cluster is still not ready or error if something went wrong.
Here you can also see the endpoint where you can connect to. In our case, it is a public IP-address of the load balancer. As described in the design section above, there are multiple ports exposed:
1 2 3 | $ kubectl get service my-cluster-router NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-cluster-router LoadBalancer 10.20.22.90 35.223.42.238 6446:30852/TCP,6447:31694/TCP,6448:31515/TCP,6449:31686/TCP 18h |
Connect to the Cluster
To connect we will need the user first. By default, there is a root user with a randomly generated password. The password is stored in the Secret object. You can always fetch the password with the following command:
1 2 | $ kubectl get secrets my-cluster-secrets -ojson | jq -r .data.root | base64 -d SomeRandomPassword |
I’m going to use port 6446, which would grant me read/write access and lead me directly to the Primary node through MySQL Router:
1 2 3 4 5 6 7 8 9 10 11 12 13 | mysql -u root -p -h 35.223.42.238 --port 6446 Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 156329 Server version: 8.0.28-19 Percona Server (GPL), Release 19, Revision 31e88966cd3 Copyright (c) 2000, 2022, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. |
Group Replication in action
Let’s see if Group Replication really works.
List the members of the cluster by running the following command:
1 2 3 4 5 6 7 8 | $ mysql -u root -p -P 6446 -h 35.223.42.238 -e 'SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;' +-------------------------------------------+--------------+-------------+ | member_host | member_state | member_role | +-------------------------------------------+--------------+-------------+ | my-cluster-mysql-0.my-cluster-mysql.mysql | ONLINE | PRIMARY | | my-cluster-mysql-1.my-cluster-mysql.mysql | ONLINE | SECONDARY | | my-cluster-mysql-2.my-cluster-mysql.mysql | ONLINE | SECONDARY | +-------------------------------------------+--------------+-------------+ |
Now we will delete one Pod (MySQL node), which also happens to have a Primary role, and see what happens:
1 2 3 4 5 6 7 8 9 10 11 | $ kubectl delete pod my-cluster-mysql-0 pod "my-cluster-mysql-0" deleted $ mysql -u root -p -P 6446 -h 35.223.42.238 -e 'SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;' +-------------------------------------------+--------------+-------------+ | member_host | member_state | member_role | +-------------------------------------------+--------------+-------------+ | my-cluster-mysql-1.my-cluster-mysql.mysql | ONLINE | PRIMARY | | my-cluster-mysql-2.my-cluster-mysql.mysql | ONLINE | SECONDARY | +-------------------------------------------+--------------+-------------+ |
One node is gone as expected. my-cluster-mysql-1 node got promoted to a Primary role. I’m still using port 6446 and the same host to connect to the database, which indicates that MySQL Router is doing its job.
After some time Kubernetes will recreate the Pod and the node will join the cluster again automatically:
1 2 3 4 5 6 7 8 | $ mysql -u root -p -P 6446 -h 35.223.42.238 -e 'SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;' +-------------------------------------------+--------------+-------------+ | member_host | member_state | member_role | +-------------------------------------------+--------------+-------------+ | my-cluster-mysql-0.my-cluster-mysql.mysql | RECOVERING | SECONDARY | | my-cluster-mysql-1.my-cluster-mysql.mysql | ONLINE | PRIMARY | | my-cluster-mysql-2.my-cluster-mysql.mysql | ONLINE | SECONDARY | +-------------------------------------------+--------------+-------------+ |
The recovery phase might take some time, depending on the data size and amount of the changes, but eventually, it will come back ONLINE:
1 2 3 4 5 6 7 8 | $ mysql -u root -p -P 6446 -h 35.223.42.238 -e 'SELECT member_host, member_state, member_role FROM performance_schema.replication_group_members;' +-------------------------------------------+--------------+-------------+ | member_host | member_state | member_role | +-------------------------------------------+--------------+-------------+ | my-cluster-mysql-0.my-cluster-mysql.mysql | ONLINE | SECONDARY | | my-cluster-mysql-1.my-cluster-mysql.mysql | ONLINE | PRIMARY | | my-cluster-mysql-2.my-cluster-mysql.mysql | ONLINE | SECONDARY | +-------------------------------------------+--------------+-------------+ |
What’s coming up next?
Some exciting capabilities and features that we are going to ship pretty soon:
- Backup and restore support for clusters with Group Replication
- We have backups and restores in the Operator, but they currently do not work with Group Replication
- Monitoring of MySQL Router in Percona Monitoring and Management (PMM)
- Even though the Operator integrates nicely with PMM, it is possible to monitor MySQL nodes only, but not MySQL Router.
- Automated Upgrades of MySQL and database components in the Operator
- We have it in all other Operators and it is just logical to add it here
Percona is an open source company and we value our community and contributors. You are greatly encouraged to contribute to Percona Software. Please read our Contributions guide and visit our community webpage.
