A customer recently asked if our Percona Operator for PostgreSQL supports the deployment of a standby cluster, which they need as part of their Disaster Recovery (DR) strategy. The answer is yes – as long as you are making use of an object storage system for backups, such as AWS S3 or GCP Cloud Storage buckets, that can be accessed by the standby cluster. In a nutshell, it works like this:
- The primary cluster is configured with pgBackRest to take backups and store them alongside archived WAL files in a remote repository;
- The standby cluster is built from one of these backups and it is kept in sync with the primary cluster by consuming the WAL files that are copied from the remote repository.
Note that the primary node in the standby cluster is not a streaming replica from any of the nodes in the primary cluster and that it relies on archived WAL files to replicate events. For this reason, this approach cannot be used as a High Availability (HA) solution. Even though the primary use of a standby cluster in this context is DR, it can be also employed for migrations as well.
So, how can we create a standby cluster using the Percona Operator for PostgreSQL? We will show you next. But first, let’s create a primary cluster for our example.
Creating a Primary PostgreSQL Cluster Using the Percona Operator
You will find a detailed procedure on how to deploy a PostgreSQL cluster using the Percona operator in our online documentation. Here we want to highlight the main steps involved, particularly regarding the configuration of object storage, which is a crucial requirement and should better be done during the initial deployment of the cluster. In the following example, we will deploy our clusters using the Google Kubernetes Engine (GKE) but you can find similar instructions for other environments in the previous link.
Considering you have a Google account configured as well as the gcloud (from the Google Cloud SDK suite) and kubectl command-line tools installed, authenticate yourself with gcloud auth login, and off we go!
Creating a GKE Cluster and Basic Configuration
The following command will create a default cluster named “cluster-1” and composed of three nodes. We are creating it in the us-central1-a zone using e2-standard-4 VMs but you may choose different options. In fact, you may also need to indicate the project name and other main settings if you do not have your gcloud environment pre-configured with them:
1 | gcloud container clusters create cluster-1 --preemptible --machine-type e2-standard-4 --num-nodes=3 --zone us-central1-a |
Once the cluster is created, use your IAM identity to control access to this new cluster:
1 | kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account) |
Finally, create the pgo namespace:
1 | kubectl create namespace pgo |
and set the current context to refer to this new namespace:
1 | kubectl config set-context $(kubectl config current-context) --namespace=pgo |
Creating a Cloud Storage Bucket
Remember for this setup we need a Google Cloud Storage bucket configured as well as a Service Account created with the necessary privileges/roles to access it. The respective procedures to obtain these vary according to how your environment is configured so we won’t be covering them here. Please refer to the Google Cloud Storage documentation for the exact steps. The bucket we created for the example in this post was named cluster1-backups-and-wals.
Likewise, please refer to the Creating and managing service account keys documentation to learn how to create a Service Account and download the corresponding key in JSON format – we will need to provide it to the operator so our PostgreSQL clusters can access the storage bucket.
Creating the Kubernetes Secrets File to Access the Storage Bucket
Create a file named my-gcs-account-secret.yaml with the following structure:
| apiVersion: v1 kind: Secret metadata: name: cluster1-backrest-repo-config type: Opaque data: gcs-key: <VALUE> |
replacing the <VALUE> placeholder by the output of the following command according to the OS you are using:
Linux:
1 | base64 --wrap=0 your-service-account-key-file.json |
macOS:
1 | base64 your-service-account-key-file.json |
Installing and Deploying the Operator
The most practical way to install our operator is by cloning the Git repository, and then moving inside its directory:
1 2 | git clone -b v1.1.0 https://github.com/percona/percona-postgresql-operator cd percona-postgresql-operator |
The following command will deploy the operator:
1 | kubectl apply -f deploy/operator.yaml |
We have already prepared the secrets file to access the storage bucket so we can apply it now:
1 | kubectl apply -f my-gcs-account-secret.yaml |
Now, all that is left is to customize the storages options in the deploy/cr.yaml file to indicate the use of the GCS bucket as follows:
| storages: my-gcs: type: gcs bucket: cluster1-backups-and-wals |
We can now deploy the primary PostgreSQL cluster (cluster1):
1 | kubectl apply -f deploy/cr.yaml |
Once the operator has been deployed, you can run the following command to do some housekeeping:
1 | kubectl delete -f deploy/operator.yaml |
Creating a Standby PostgreSQL Cluster Using the Percona Operator
After this long preamble, let’s look at what brought you here: how to deploy a standby cluster, which we will refer to as cluster2, that will replicate from the primary cluster.
Copying the Secrets Over
Considering you probably have customized the passwords you use in your primary cluster and that they differ from the default values found in the operator’s git repository, we need to make a copy of the secrets files, adjusted to the standby cluster’s name. The following procedure facilitates this task, saving the secrets files under /tmp/cluster1-cluster2-secrets (you can choose a different target directory):
| NOTE: make sure you have the yq tool installed in your system. |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | mkdir -p /tmp/cluster1-cluster2-secrets/ export primary_cluster_name=cluster1 export standby_cluster_name=cluster2 export secrets="${primary_cluster_name}-users" kubectl get secret/$secrets -o yaml | yq eval 'del(.metadata.creationTimestamp)' - | yq eval 'del(.metadata.uid)' - | yq eval 'del(.metadata.selfLink)' - | yq eval 'del(.metadata.resourceVersion)' - | yq eval 'del(.metadata.namespace)' - | yq eval 'del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' - | yq eval '.metadata.name = "'"${secrets/$primary_cluster_name/$standby_cluster_name}"'"' - | yq eval '.metadata.labels.pg-cluster = "'"${standby_cluster_name}"'"' - >/tmp/cluster1-cluster2-secrets/${secrets/$primary_cluster_name/$standby_cluster_name} |
Deploying the Standby Cluster: Fast Mode
Since we have already covered the procedure used to create the primary cluster in detail in a previous section, we will be presenting the essential steps to create the standby cluster below and provide additional comments only when necessary.
| NOTE: the commands below are issued from inside the percona-postgresql-operator directory hosting the git repository for our operator. |
Deploying a New GKE Cluster Named cluster-2
This time using the us-west1-b zone here:
1 2 3 4 5 | gcloud container clusters create cluster-2 --preemptible --machine-type e2-standard-4 --num-nodes=3 --zone us-west1-b kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value core/account) kubectl create namespace pgo kubectl config set-context $(kubectl config current-context) --namespace=pgo kubectl apply -f deploy/operator.yaml |
Apply the Adjusted Kubernetes Secrets:
1 2 3 | export standby_cluster_name=cluster2 export secrets="${standby_cluster_name}-users" kubectl create -f /tmp/cluster1-cluster2-secrets/$secrets |
The list above does not include the GCS secret file; the key contents remain the same but the backrest-repo pod name needs to be adjusted. Make a copy of that file:
1 | cp my-gcs-account-secret.yaml my-gcs-account-secret-2.yaml |
then edit the copy to indicate “cluster2-” instead of “cluster1-”:
| name: cluster2-backrest-repo-config |
You can apply it now:
1 | kubectl apply -f my-gcs-account-secret-2.yaml |
The cr.yaml file of the Standby Cluster
Let’s make a copy of the cr.yaml file we customized for the primary cluster:
1 | cp deploy/cr.yaml deploy/cr-2.yaml |
and edit the copy as follows:
1) Change all references (that are not commented) from cluster1 to cluster2 – including current-primary but excluding the bucket reference, which in our example is prefixed with “cluster1-”; the storage section must remain unchanged. (We know it’s not very practical to replace so many references, we still need to improve this part of the routine).
2) Enable the standby option:
| standby: true |
3) Provide a repoPath that points to the GCS bucket used by the primary cluster (just below the storages section, which should remain the same as in the primary cluster’s cr.yaml file):
| repoPath: “/backrestrepo/cluster1-backrest-shared-repo” |
And that’s it! All that is left now is to deploy the standby cluster:
1 | kubectl apply -f deploy/cr-2.yaml |
With everything working on the standby cluster, do some housekeeping:
1 | kubectl delete -f deploy/operator.yaml |
Verifying it all Works as Expected
Remember that the standby cluster is created from a backup and relies on archived WAL files to be continued in sync with the primary cluster. If you make a change in the primary cluster, such as adding a row to a table, that change won’t reach the standby cluster until the WAL file it has been recorded to is archived and consumed by the standby cluster.
When checking if all is working with the new setup, you can force the rotation of the WAL file (and subsequent archival of the previous one) in the primary node of the primary cluster to accelerate the sync process by issuing:
1 | psql> SELECT pg_switch_wal(); |
The Percona Operators automate the creation, alteration, or deletion of members in your Percona Distribution for MySQL, MongoDB, or PostgreSQL environment.
Are there any plans to implement PostgreSQL physical replication with PGO?
Hi Markus, what exactly do you mean? The standby cluster is indeed using physical replication. It’s just not streaming changes directly from the primary cluster. However, within the same cluster, nodes are using streaming replication, which can optionally be synchronous.