GTID replication has made it convenient to setup and maintain MySQL replication. You need not worry about binary log file and position thanks to GTID and auto-positioning. However, things can go wrong when pointing a slave to a different master. Consider a situation where the new master has executed transactions that haven’t been executed on the old master. If the corresponding binary logs have been purged already, how do you point the slave to the new master?
The scenario
Based on technical requirements and architectural change, there is a need to point the slave to a different master by
- Pointing it to another node in a PXC cluster
- Pointing it to another master in master/master replication
- Pointing it to another slave of a master
- Pointing it to the slave of a slave of the master … and so on and so forth.
Theoretically, pointing to a new master with GTID replication is easy. All you have to do is run:
1 2 3 4 | STOP SLAVE; CHANGE MASTER TO MASTER_HOST='new_master_ip'; START SLAVE; SHOW SLAVE STATUS\G |
Alas, in some cases, replication breaks due to missing binary logs:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | *************************** 1. row *************************** Slave_IO_State: Master_Host: pxc_57_5 Master_User: repl Master_Port: 3306 **redacted** Slave_IO_Running: No Slave_SQL_Running: Yes ** redacted ** Last_IO_Errno: 1236 Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires.' ** redacted ** Master_Server_Id: 1 Master_UUID: 4998aaaa-6ed5-11e8-948c-0242ac120007 Master_Info_File: /var/lib/mysql/master.info ** redacted ** Last_IO_Error_Timestamp: 180613 08:08:20 Last_SQL_Error_Timestamp: ** redacted ** Retrieved_Gtid_Set: Executed_Gtid_Set: 1904cf31-912b-ee17-4906-7dae335b4bfc:1-3 Auto_Position: 1 |
The strange issue here is that if you point the slave back to the old master, replication works just fine. The error says that there are missing binary logs in the new master that the slave needs. If there’s no problem with replication performance and the slave can easily catch up, then it looks like there are transactions executed in the new master that have not been executed in the old master but are recorded in the missing binary logs. The binary logs are most likely lost due to manually purging with PURGE BINARY LOGS or automatic purging if expire_logs_days is set.
At this point, it would be prudent to check and sync old master and new master with tools such as pt-table-checksum and pt-table-sync. However, if a consistency check has been performed and no differences have been found, or there’s confidence that the new master is a good copy—such as another node in the PXC cluster—you can follow the steps below to resolve the problem.
Solution
To solve the problem, the slave needs to execute the missing transactions. But since these transactions have been purged, the steps below provide the workaround.
Step 1 Find the GTID sequences that are purged from the new master that is needed by the slave
To identify which GTID sequences are missing, run SHOW GLOBAL VARIABLES LIKE 'gtid_purged'; and SHOW MASTER STATUS; on the new master and SHOW GLOBAL VARIABLES LIKE 'gtid_executed'; on the slave:
New Master:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | mysql> SHOW GLOBAL VARIABLES LIKE 'gtid_purged'; +---------------+-------------------------------------------------------------------------------------+ | Variable_name | Value | +---------------+-------------------------------------------------------------------------------------+ | gtid_purged | 1904cf31-912b-ee17-4906-7dae335b4bfc:1-2, 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 | +---------------+-------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) mysql> SHOW MASTER STATUS; +------------------+----------+--------------+------------------+-------------------------------------------------------------------------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +------------------+----------+--------------+------------------+-------------------------------------------------------------------------------------+ | mysql-bin.000004 | 741 | | | 1904cf31-912b-ee17-4906-7dae335b4bfc:1-6, 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 | +------------------+----------+--------------+------------------+-------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) |
Slave:
1 2 3 4 5 6 7 | mysql> SHOW GLOBAL VARIABLES LIKE 'gtid_executed'; +---------------+------------------------------------------+ | Variable_name | Value | +---------------+------------------------------------------+ | gtid_executed | 1904cf31-912b-ee17-4906-7dae335b4bfc:1-3 | +---------------+------------------------------------------+ 1 row in set (0.00 sec) |
Take note that 1904cf31-912b-ee17-4906-7dae335b4bfc and 1904cf31-912b-ee17-4906-7dae335b4bfc are UUIDs and refer to the MySQL instance where the transaction originated from.
Based on the output:
- The slave has executed 1904cf31-912b-ee17-4906-7dae335b4bfc:1-3
- The new master has executed 1904cf31-912b-ee17-4906-7dae335b4bfc:1-6 and 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11
- The new master has purged 1904cf31-912b-ee17-4906-7dae335b4bfc:1-2 and 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11
This means that the slave has no issue with 1904cf31-912b-ee17-4906-7dae335b4bfc it requires sequences 4-6 and sequences 3-6 are still available in the master. However, the slave cannot fetch sequences 1-11 from 4998aaaa-6ed5-11e8-948c-0242ac120007 because these has been purged from the master.
To summarize, the missing GTID sequences are 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11.
Step 2: Identify where the purged GTID sequences came from
From the SHOW SLAVE STATUS output in the introduction section, it says that the Master_UUID is 4998aaaa-6ed5-11e8-948c-0242ac120007, which means the new master is the source of the missing transactions. You can also verify the new Master’s UUID by running SHOW GLOBAL VARIABLES LIKE 'server_uuid';
1 2 3 4 5 6 7 | mysql> SHOW GLOBAL VARIABLES LIKE 'server_uuid'; +---------------+--------------------------------------+ | Variable_name | Value | +---------------+--------------------------------------+ | server_uuid | 4998aaaa-6ed5-11e8-948c-0242ac120007 | +---------------+--------------------------------------+ 1 row in set (0.00 sec) |
If the new master’s UUID does not match the missing GTID, it is most likely that this missing sequence came from its old master, another master higher up the chain or from another PXC node. If that other master still exists, you can run the same query on those masters to check.
The missing sequences are small such as 1-11. Typically, commands executed locally are due to performing maintenance on this server directly. For example, creating users, fixing privileges or updating passwords. However, you have no guarantee that this is the reason, since the binary logs have already been purged. If you still want to point the slave to the new master, proceed to step 3 or step 4.
Step 3. Injecting the missing transactions on the slave with empty transactions
The workaround is to pretend that those missing GTID sequences have been executed on the slave by injecting 11 empty transactions as instructed here by running:
1 2 3 | SET GTID_NEXT='UUID:SEQUENCE_NO'; BEGIN;COMMIT; SET GTID_NEXT='AUTOMATIC'; |
It looks tedious, but a simple script can automate this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 | cat empty_transaction_generator.sh #!/bin/bash uuid=$1 first_sequence_no=$2 last_sequence_no=$3 while [ "$first_sequence_no" -le "$last_sequence_no" ] do echo "SET GTID_NEXT='$uuid:$first_sequence_no';" echo "BEGIN;COMMIT;" first_sequence_no=`expr $first_sequence_no + 1` done echo "SET GTID_NEXT='AUTOMATIC';" bash empty_transaction_generator.sh 4998aaaa-6ed5-11e8-948c-0242ac120007 1 11 SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:1'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:2'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:3'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:4'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:5'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:6'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:7'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:8'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:9'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:10'; BEGIN;COMMIT; SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:11'; BEGIN;COMMIT; SET GTID_NEXT='AUTOMATIC'; |
Before executing the generated output on the slave, stop replication first:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | mysql> STOP SLAVE; Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:1'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:2'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:3'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:4'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:5'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:6'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:7'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:8'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.01 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:9'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:10'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='4998aaaa-6ed5-11e8-948c-0242ac120007:11'; Query OK, 0 rows affected (0.00 sec) mysql> BEGIN;COMMIT; Query OK, 0 rows affected (0.00 sec) Query OK, 0 rows affected (0.00 sec) mysql> SET GTID_NEXT='AUTOMATIC'; Query OK, 0 rows affected (0.00 sec) |
There’s also an even easier solution of injecting empty transactions by using mysqlslavetrx from MySQL utilities. By stopping the slave first and running
mysqlslavetrx --gtid-set=4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 --slaves=root:password@:3306 you will achieve the same result as above.
By running SHOW GLOBAL VARIABLES LIKE 'gtid_executed'; on the slave you can see that sequences 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 have been executed already:
1 2 3 4 5 6 7 8 | mysql> SHOW GLOBAL VARIABLES LIKE 'gtid_executed'; +---------------+-------------------------------------------------------------------------------------+ | Variable_name | Value | +---------------+-------------------------------------------------------------------------------------+ | gtid_executed | 1904cf31-912b-ee17-4906-7dae335b4bfc:1-3, 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 | +---------------+-------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) |
Resume replication and check if replication is healthy by running START SLAVE; and SHOW SLAVE STATUS\G
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | mysql> START SLAVE; Query OK, 0 rows affected (0.01 sec) mysql> SHOW SLAVE STATUS\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: pxc_57_5 Master_User: repl Master_Port: 3306 ** redacted ** Slave_IO_Running: Yes Slave_SQL_Running: Yes ** redacted ** Seconds_Behind_Master: 0 ** redacted ** Master_Server_Id: 1 Master_UUID: 4998aaaa-6ed5-11e8-948c-0242ac120007 ** redacted ** Retrieved_Gtid_Set: 1904cf31-912b-ee17-4906-7dae335b4bfc:4-6 Executed_Gtid_Set: 1904cf31-912b-ee17-4906-7dae335b4bfc:1-6, 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 Auto_Position: 1 ** redacted ** 1 row in set (0.00 sec) |
At this point, we have already solved the problem. However, there’s another way to restore the slave much faster but at the cost of erasing all the existing binary logs on the slave as mentioned in this article. If you want to do this, proceed to step 4.
Step 4. Add the missing sequences to GTID_EXECUTED by modifying GTID_PURGED.
CRITICAL NOTE:
If you followed the steps in Step 3, you do not need to perform Step 4!
To add the missing transactions, you’ll need to stop the slave, reset the master, place the original value of gtid_executed and the missing sequences in gtid_purged variable. A word of caution on using this method: this will purge the existing binary logs of the slave.
1 2 3 4 5 6 | mysql> STOP SLAVE; Query OK, 0 rows affected (0.02 sec) mysql> RESET MASTER; Query OK, 0 rows affected (0.02 sec) mysql> SET GLOBAL gtid_purged="1904cf31-912b-ee17-4906-7dae335b4bfc:1-3,4998aaaa-6ed5-11e8-948c-0242ac120007:1-11"; Query OK, 0 rows affected (0.02 sec) |
Similar to Step 3, running SHOW GLOBAL VARIABLES LIKE 'gtid_executed'; on the slave shows that sequence 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 has been executed already:
1 2 3 4 5 6 7 8 | mysql> SHOW GLOBAL VARIABLES LIKE 'gtid_executed'; +---------------+-------------------------------------------------------------------------------------+ | Variable_name | Value | +---------------+-------------------------------------------------------------------------------------+ | gtid_executed | 1904cf31-912b-ee17-4906-7dae335b4bfc:1-3, 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 | +---------------+-------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) |
Run START SLAVE; and SHOW SLAVE STATUS\G to resume replication and check if replication is healthy:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | mysql> START SLAVE; Query OK, 0 rows affected (0.01 sec) mysql> SHOW SLAVE STATUS\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: pxc_57_5 Master_User: repl Master_Port: 3306 ** redacted ** Slave_IO_Running: Yes Slave_SQL_Running: Yes ** redacted ** Seconds_Behind_Master: 0 ** redacted ** Master_Server_Id: 1 Master_UUID: 4998aaaa-6ed5-11e8-948c-0242ac120007 ** redacted ** Retrieved_Gtid_Set: 1904cf31-912b-ee17-4906-7dae335b4bfc:4-6 Executed_Gtid_Set: 1904cf31-912b-ee17-4906-7dae335b4bfc:1-6, 4998aaaa-6ed5-11e8-948c-0242ac120007:1-11 Auto_Position: 1 ** redacted ** 1 row in set (0.00 sec) |
Step 5. Done
Summary
In this article, I demonstrated how to point the slave to a new master even if it’s missing some binary logs that need to be executed. Although, it is possible to do so with the workarounds shared above, it is prudent to check the consistency of the old and new master first before switching the slave to the new master.
What about data loss?