Last Resort: How to Use a Backup to Start a Secondary Instance for MongoDB

In this blog post, I’ll look at how you can use a backup to start a secondary instance for MongoDB.

Although the documentation says it is not possible to use a backup to start a secondary, sometimes this is the only possible way to start a new instance. In this blog post, we will explain how to bypass this limitation and use a backup to start a secondary instance.

The initial sync/rsync or snapshot works fine when the instances are in the same data center, but it can fail or be really slow. Much slower than moving a compressed backup between data centers.

Not every backup can be used as a source for starting a replica set. The backup must have the oplog file. This means the backup must be done in a previously existent replica set using the --oplog flag point in time backup when dumping the collections. The time spent to move and restore the backup file must be less than the oplog window.

Please follow the next steps to create a secondary from a backup:

1. Create a backup using the --oplog command.
   
   Shell

   mongodump --oplog -o /tmp/backup/

   1	mongodump --oplog -o /tmp/backup/

2. Backup the replica set collection from the local database.
   
   
   Shell

   mongodump -d local -c system.replset

   1	mongodump -d local -c system.replset

3. After backup finishes please confirm the oplog.rs file is in the backup folder.
4. Use bsondump to convert the oplog to JSON. We will use the last oplog entry as a starting point for the replica set.
   
   Shell

   bsondump oplog.rs > oplog.rs.txt

   1	bsondump oplog.rs > oplog.rs.txt

5. Initiate the new instance without the replica set parameter configured. At this point, the instance will act as a single instance.
6. Restore the database normally using --oplogreplay to apply all the oplogs that have been recorded while the backup was running.
   
   Shell

   mongorestore /tmp/backup --oplogReplay

   1	mongorestore /tmp/backup --oplogReplay

7. Connect to this server and use the local database to create the oplog.rs collection. Please use the same value as the other members (e.g., 20 GB).
   
   Shell

   mongo use local db.runCommand( { create: "oplog.rs", capped: true, size: (20 * 1024 * 1024 * 1024) } )

   1 2 3

   mongo use local db.runCommand( { create: "oplog.rs", capped: true, size: (20 * 1024 * 1024 * 1024) } )

8. From the oplog.rs.txt generated in step 4, get the last line and copy the fields ts and h values to a MongoDB document.
   
   
   Shell

   tail -n 1 opslog.rs.txt

   1	tail -n 1 opslog.rs.txt

9. Insert the JSON value to the oplog.rs collection that was created before.
   
   Shell

   mongo use local db.oplog.rs.insert({ ts: Timestamp(12354454,1), h: NumberLong("-3434387384732843")})

   1 2 3

   mongo use local db.oplog.rs.insert({ ts: Timestamp(12354454,1), h: NumberLong("-3434387384732843")})

10. Restore the replset collection to the local database.
    
    Shell

    mongorestore -d local -c system.replset ./backup/repliset.bson

    1	mongorestore -d local -c system.replset ./backup/repliset.bson

11. Stop the service and edit the parameter replica set name to match the existing replica set.
12. Connect to the primary and add this new host. The new host must start catching up the oplog and get in sync after a few hours/minutes, depending on the number of operations the replica set handles. It is important to consider adding this new secondary as a hidden secondary, without votes if possible, to avoid triggering an election. When the secondary is added to the replica set drivers, it will start using this host to perform reads. If you don’t add the server with hidden: true, the application will read inconsistent data (old data).
    
    
    Shell

    mongo PRIMARY>rs.add({_id : <your id>, host: "<newhost:27017>", hidden : true, votes : 0, priority :0})

    1 2	mongo PRIMARY>rs.add({_id : <your id>, host: "<newhost:27017>", hidden : true, votes : 0, priority :0})

13. Please check the replication lag, and once the seconds behind master is near to zero, change the host parameters in the replica set to hidden: false and priority or votes.
    
    Shell

    mongo PRIMARY> rs.printSlaveReplicationInfo()

    1 2	mongo PRIMARY> rs.printSlaveReplicationInfo()

14. We are considering a replica set with three members, where the new secondary has the ID 2 in the member’s array. Use the following command to unhide the secondary and make it available for reads. The priority and votes depend on your environment. Please notice you might need to change the member ID.
    
    Shell

    mongo PRIMARY> cfg = rs.config() cfg.members[2].hidden = false cfg.members[2].votes = 1 cfg.members[2].priority = 1 PRIMARY> rs.reconfig(rs)

    1 2 3 4 5 6

    mongo PRIMARY> cfg = rs.config() cfg.members[2].hidden = false cfg.members[2].votes = 1 cfg.members[2].priority = 1 PRIMARY> rs.reconfig(rs)

I hope this tutorial helps in an emergency situation. Please consider using initial sync, disk snapshots and hot backups before using this method.

Feel free to reach out me on twitter @AdamoTonete or @percona.