The practice of running databases in containers continues to grow in popularity. As a Technical Account Manager at Percona, I get asked regularly about our Percona Distribution for MySQL Operator. Additionally, I’m asked what I’m seeing in the industry in terms of adoption. In most cases, the questions stem around new deployments. Our DBaaS tool (currently in Technical Preview) makes launching a new cluster in a Kubernetes deployment trivial.
Once the operator completes and verifies the setup, the UI displays the endpoint and credentials and you are on your way. Voila! You now have a cluster, behind a load balancer, that you can access from within your k8s cluster or externally:
This is all well and good, giving developers the opportunity to quickly build new applications. However, a new question recently came up in a client call: how can I migrate an existing database into Kubernetes? This got me thinking, so I decided to spin up some test servers and experiment with the backup and restore functionality of the operator.
General Migration Process
For my testing, I wanted to follow the standard process:
- Take a binary snapshot of the primary server
- Restore that snapshot into the new cluster
- Start replication from the primary server to the new cluster
- Cutover the application after replication is caught up
Looking through the operator documentation, I wanted to leverage the existing restore functionality in my testing. Through some manual updates to the service created by the DBaaS tool, I defined my S3 credentials and details:
1 | kubectl edit perconaxtradbcluster.pxc.percona.com/mbenshoof-pxc-cluster |
Next, I verified that I could take and restore backups from the command line (using the kubectl tool). So far, so good.
Finally, it was time to test the main step of a generic migration. I had a test server set up and took a streaming backup (using xbcloud with the –md5 switch passed) directly into the S3 bucket configured in the operator. The final step was to define a custom restore resource and kick off the restore:
1 2 3 4 5 6 7 8 9 10 11 12 | # cat restore-external.yaml apiVersion: pxc.percona.com/v1 kind: PerconaXtraDBClusterRestore metadata: name: restore-external-v1 spec: pxcCluster: migrate-from-external backupSource: destination: s3://<S3-BUCKET-FOR-BACKUPS>/external-80-full_backup s3: credentialsSecret: <MY-K8S-AWS-SECRET> region: us-east-1 |
After some time, I verified that the restoration succeeded and also verified that my DBaaS cluster was operational again:
1 | $ kubectl logs job/restore-job-restore-external-v1-migrate-from-external |
If everything is fine, you can clean up the job:
1 2 3 4 5 | $ kubectl delete pxc-restore/restore-external-v1 Completed: 2021-06-04T16:42:58Z State: Succeeded Events: <none> |
Success! I was able to restore my external 8.0 instance into a new DBaaS-generated cluster. The final step in a migration process is to set up replication into the new cluster.
Setting up Replication
With a replication user-defined, it was trivial to get replication running into the new cluster. In fact, there was no difference between this test and setting up replication normally. Once I allowed access to the source server (via VPC Security Groups), I simply started replication and verified that I was picking up new transactions:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 | mysql> show replica status\G *************************** 1. row *************************** Replica_IO_State: Waiting for master to send event Source_Host: 10.1.1.226 Source_User: repl Source_Port: 3306 Connect_Retry: 60 Source_Log_File: binlog.000005 Read_Source_Log_Pos: 679 Relay_Log_File: migrate-from-external-pxc-0-relay-bin.000002 ... Seconds_Behind_Source: 0 ... Retrieved_Gtid_Set: c6a4c262-c4a7-11eb-bed8-0e0d8750f515:84-85 Executed_Gtid_Set: c6a4c262-c4a7-11eb-bed8-0e0d8750f515:1-85 Auto_Position: 1 ... 1 row in set (0.00 sec) |
Just to ensure things were working as expected, I ran a few queries and sent some write traffic to my primary test server. Everything worked as expected when setting up replication from a stand-alone instance into the new cluster!
Primary Challenge – User Credentials
Working through this exercise, I did identify the main limitation in this process. When I first attempted this exercise, the restore process hung and eventually failed. The reason: conflicting user credentials in the freshly restored cluster.
The operator manages the cluster nodes via credentials defined in a Kubernetes secret. When new clusters are created, random and unique credentials are generated and used to set up the cluster. While this is great for a new cluster and keeps it secure upon launch, it isn’t ideal for the migration process.
While playing around with various migration test variations, I used one of the following two workaround methods:
- Modifying the source database with the newly generated cluster credentials
- Modifying the k8s secret file with credentials from the existing database
Modifying the Source Database
For this method, the workaround I used was capturing all of the operator generated credentials on the empty cluster via pt-show-grants:
1 | pt-show-grants --only=clustercheck,monitor,operator,proxyadmin,root,xtrabackup |
I then applied these grants to the source server (in practice, I would never do this, but this was just an exercise with dummy data). Once the credentials were updated and confirmed, I then took the S3 streaming backup and the restore went off without a hitch.
Modifying the Secrets File
For the second workaround, there are some assumptions:
- You know all the users created by the operator by default
- Those users exist (or can be created) on the source database
In this workaround, I grabbed the password(s) from the existing source cluster as the first step. Once I had them base64 encoded, it was just a matter of editing the secrets created with the new cluster. I only needed to modify the dbaas-* generated secrets as that was kept in sync with all the other secret files.
The main thing in the process – ensure the secrets file is updated BEFORE kicking off the restore job. Secrets are only reloaded when pods are terminated and restarted, so doing that cleanly is important. The restore job does a few things:
- Pull the backup from S3 into a new container
- Prepare and validate the backup
- Stop the old cluster
- Restart the cluster with the freshly prepared backup
Assuming you have the secrets file in place before starting the job, the new credentials will be picked up nicely when the cluster restarts and all will go well. Otherwise, you’ll be looking at lots of manual battles and debugging.
Other Limitations
The other primary limitation with the current release is that the source needs to also be running 8.0. As the restore procedure uses xtrabackup-80 for the process, providing a 5.7 backup will result in the following error:
1 | This version of Percona XtraBackup can only perform backups and restores against MySQL 8.0 and Percona Server 8.0, please use Percona Xtrabackup 2.4 for this database. |
To handle a 5.7 -> 8.0 migration, you would need to do a logical dump/reload of the databases in question. Like the binary restore, the user credentials will be problematic so it would be preferable to omit the user tables. This process comes with the standard 5.7 -> 8.0 challenges, but it is possible.
Summary
As this process is not supported (or even recommended) in the current release, I’ve omitted some details and configuration from this post. Despite the challenges and the process being a little rough around the edges, it was great to validate xtrabackup restoration and replication into k8s from an external source.
I’m excited to see the progress in the coming months as our DBaaS offering approaches GA. With features like UX management of the process and MySQL user migration on the roadmap, it should be much easier in the future to migrate existing databases into Kubernetes.
With the investments we are seeing around large k8s deployments, it would be a shame to limit it to new applications. As the Percona Distribution for MySQL Operator continues to evolve along with the DBaaS interface, the options should be unlimited. As always, if you need help evaluating an existing architecture or migrating into Kubernetes, don’t hesitate to reach out to our Professional Services team!
