How to Change MongoDB Storage Engines Without Downtime

This blog is another in the series for the Percona Server for MongoDB 3.4 bundle release. Today’s blog post is about how to migrate between Percona Server for MongoDB storage engines without downtime.

Today, the default storage engine for MongoDB is WiredTiger. In previous versions (before 3.2), it was MMAPv1.

Percona Server for MongoDB features some additional storage engines, giving you the freedom for a DBA to choose the best storage based on application workload. Our storages engines are:

• MMAPv1
• WiredTiger
• RocksDB
• In Memory

By design, each storage engine has its own algorithm and disk usage patterns. We simply stop and start Percona Server for MongoDB using different storage engines.

There are two common methods to change storage engines. One requires downtime, and the second doesn’t.

All the database operations are the same, even if it is using a different storage engine. From the database perspective, it doesn’t matter what storage engine gets used. The database layer asks the persistence API to save or retrieve data regardless.

For a single database instance, the best storage engine migration method is to start replication and add a secondary node with a different storage engine. Then stepdown() the primary, making the secondary the new primary (killing the old primary).

However, this isn’t always an option. In this case, create a backup and use the backup to restore the database.

In the following set of steps, we’ll explain how to migrate a replica set storage engine from WiredTiger to RocksDB without downtime. I’m assuming that the replica set is already configured and doesn’t have any replication lag.

Please follow the instructions below:

1. Check replica set status and identify the primary and secondaries. (Part of the output has been hidden to make it easier to read.):
   
   foo:PRIMARY> rs.status() { "set" : "foo", "date" : ISODate("2017-02-18T18:47:54.349Z"), "myState" : 2, "term" : NumberLong(2), "syncingTo" : "adamo-percona:27019", "heartbeatIntervalMillis" : NumberLong(2000), "members" : [ { "_id" : 0, "name" : "test:27017", "stateStr" : "PRIMARY" (...) }, { "_id" : 1, "name" : "test:27018", "stateStr" : "SECONDARY" (...) }, { "_id" : 2, "name" : "test:27019", "stateStr" : "SECONDARY" (...) } { ... } ], "ok" : 1 }

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

   foo:PRIMARY> rs.status() { "set" : "foo", "date" : ISODate("2017-02-18T18:47:54.349Z"), "myState" : 2, "term" : NumberLong(2), "syncingTo" : "adamo-percona:27019", "heartbeatIntervalMillis" : NumberLong(2000), "members" : [ { "_id" : 0, "name" : "test:27017", "stateStr" : "PRIMARY" (...) }, { "_id" : 1, "name" : "test:27018", "stateStr" : "SECONDARY" (...) }, { "_id" : 2, "name" : "test:27019", "stateStr" : "SECONDARY" (...) } { ... } ], "ok" : 1 }

2. Choose the secondary for the new storage engine, and change its priority to 0:
   
   
   foo:PRIMARY> cfg = rs.config()

   1	foo:PRIMARY> cfg = rs.config()

   
   We are going to work with test:27018 and test:27019. They are respectively the index 1 and 2 in the array members.
   
3. Change the last secondary to the first instance to replace the storage engine:
   
   
   foo:PRIMARY> cfg.members[2].name test:27019 foo:PRIMARY> cfg.members[2].priority = 0 0 foo:PRIMARY> cfg.members[2].hidden = true true foo:PRIMARY> rs.reconfig(cfg) { "ok" : 1 }

   1 2 3 4 5 6 7 8 9

   foo:PRIMARY> cfg.members[2].name test:27019   foo:PRIMARY> cfg.members[2].priority = 0 0 foo:PRIMARY> cfg.members[2].hidden = true true foo:PRIMARY> rs.reconfig(cfg) { "ok" : 1 }

4. Check if the configuration is in place:
   
   foo:PRIMARY>rs.config() { "_id" : "foo", "version" : 4, "protocolVersion" : NumberLong(1), "members" : [ { "_id" : 0, "host" : "test:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "votes" : 1 }, { "_id" : 1, "host" : "test:27018", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 2, "host" : "test:27019", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : true, <-- "priority" : 0, <-- "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : {...} }

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

   foo:PRIMARY>rs.config() { "_id" : "foo", "version" : 4, "protocolVersion" : NumberLong(1), "members" : [ { "_id" : 0, "host" : "test:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "votes" : 1 }, { "_id" : 1, "host" : "test:27018", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 2, "host" : "test:27019", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : true, <-- "priority" : 0, <-- "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : {...} }

5. Then stop the desired secondary and wipe the database folder. As we are running the replica set in a testing box, I’m going to kill the process running on port 27019. If using services please run: sudo service mongod stop on the secondary box. Before starting the mongod service, add the --storageEngine parameter to the config file or application parameter:
   
   ps -ef | grep mongodb | grep 27019 kill < mongod pid>; rm -rf /data3/* ./mongod --dbpath data3 --logpath data3/log3.log --fork --port 27019 <strong>--storageEngine=rocksdb</strong> --replSet foo

   1 2 3 4

   ps -ef | grep mongodb | grep 27019 kill < mongod pid>; rm -rf /data3/* ./mongod --dbpath data3 --logpath data3/log3.log --fork --port 27019 <strong>--storageEngine=rocksdb</strong> --replSet foo

   
   
   <config file> storage: engine: rocksdb

   1 2 3

   <config file> storage:   engine: rocksdb

6. This instance is now using the RocksDB storage engine and will perform an initial sync. When it finishes, to get the data from the primary node remove the hidden = false flag and let the application query this box:
   
   foo:PRIMARY> cfg = rs.config() foo:PRIMARY> cfg.members[2].hidden = false false foo:PRIMARY> rs.reconfig(cfg) { "ok" : 1 }

   1 2 3 4 5

   foo:PRIMARY> cfg = rs.config() foo:PRIMARY> cfg.members[2].hidden = false false foo:PRIMARY> rs.reconfig(cfg) { "ok" : 1 }

7. Repeat step 6 for box test:27018, and use the following command as step 6. This makes one of the secondaries become the primary. Please be sure all secondaries are healthy before proceeding:
   
   foo:PRIMARY> cfg = rs.config() foo:PRIMARY> cfg.members[2].hidden = false false foo:PRIMARY> cfg.members[2].priority = 1 foo:PRIMARY> cfg.members[1].priority = 1

   1 2 3 4 5

   foo:PRIMARY> cfg = rs.config() foo:PRIMARY> cfg.members[2].hidden = false false foo:PRIMARY> cfg.members[2].priority = 1 foo:PRIMARY> cfg.members[1].priority = 1

8. When both secondaries are available for reading and in sync with the primary, we need to change the primary’s storage engine. To do so, please perform a stepdown() in the primary, making this instance secondary. An election is triggered (and may take a few seconds to complete):
   
   foo:PRIMARY> rs.stepDown() 2017-02-20T16:34:53.814-0300 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:27019' : DB.prototype.runCommand@src/mongo/shell/db.js:135:1 DB.prototype.adminCommand@src/mongo/shell/db.js:153:16 rs.stepDown@src/mongo/shell/utils.js:1182:12 @(shell):1:1 2017-02-20T16:34:53.815-0300 I NETWORK [thread1] trying reconnect to 127.0.0.1:27019 (127.0.0.1) failed 2017-02-20T16:34:53.816-0300 I NETWORK [thread1] reconnect 127.0.0.1:27019 (127.0.0.1) ok foo:SECONDARY> rs.status()

   1 2 3 4 5 6 7 8 9 10

   foo:PRIMARY> rs.stepDown() 2017-02-20T16:34:53.814-0300 E QUERY [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:27019' : DB.prototype.runCommand@src/mongo/shell/db.js:135:1 DB.prototype.adminCommand@src/mongo/shell/db.js:153:16 rs.stepDown@src/mongo/shell/utils.js:1182:12 @(shell):1:1 2017-02-20T16:34:53.815-0300 I NETWORK [thread1] trying reconnect to 127.0.0.1:27019 (127.0.0.1) failed 2017-02-20T16:34:53.816-0300 I NETWORK [thread1] reconnect 127.0.0.1:27019 (127.0.0.1) ok   foo:SECONDARY> rs.status()

9. Please identify the new primary with rs.status() and repeat the step 5 and 7 with the old primary.

After this process, the instances will run RocksDB without experiencing downtime (just an election to change the primary).

Please feel free to ping us on Twitter @percona with any questions and suggestions for this blog post.

1 Comment

Oldest

Newest Most Voted

Inline Feedbacks

View all comments

Jerwin Roy

6 years ago

But we need to have a enough oplog size configured for the initial sync process to complete.

0