In this article series, we will talk about the basic high availability architecture of a MongoDB: the MongoDB replica set.
- Part 1 : We introduced basic replica set concepts, how it works and what its main features
- Part 2 (this post): We’ll provide a step-by-step guide to configure a three-node replica set
- Part 3: We’ll talk about how to configure transport encryption between the nodes
In part 1 we introduced and described the main features of a MongoDB replica set. In this post, we are going to present a step-by-step guide to deploy a basic and fully operational 3-nodes replica set. We’ll use just regular members, all with priority=1, no arbiter, no hidden or delayed nodes.
The environment
Our example environment is 3 virtual hosts with Ubuntu 16.04 LTS, although the configuration is the same with CentOS or other Linux distributions.
We have installed Percona Server for MongoDB on each node. Hostnames and IPs are:
- psmdb1 : 192.168.56.101
- psmdb2 : 192.168.56.102
- psmdb3 : 192.168.56.103
It is not the goal of this post to provide installation details, but in case you need them you can follow this guide: https://www.percona.com/doc/percona-server-for-mongodb/LATEST/install/index.html MongoDB installation from the repository is very easy.
Connectivity
Once we have all the nodes with MongoDB installed, we just need to be sure that each one is accessible by all the others on port 27017, the default port.
Since our members are on the same network we can simply try to test the connectivity between each pair of nodes, connecting the mongo client from one node to each of the others.
1 2 3 4 5 6 | psmdb1> mongo --host 192.168.56.102 --port 27017 psmdb1> mongo --host 192.168.56.103 --port 27017 psmdb2> mongo --host 192.168.56.101 --port 27017 psmdb2> mongo --host 192.168.56.103 --port 27017 psmdb3> mongo --host 192.168.56.101 --port 27017 psmdb3> mongo --host 192.168.56.102 --port 27017 |
If the mongo client is not able to connect, we need to check the network configuration, or to configure or disable the firewall.
Hostnames
Configuring the hostnames into our hosts is not mandatory for the replica set. In fact you can configure the replica set using just the IPs and it’s fine. But we need to define the hostnames because they will be very useful when we discuss how to configure internal encryption in Part 3.
We need to ensure that each member is accessible by way of resolvable DNS or hostnames.
Set up each node in the /etc/hosts file
1 2 3 4 5 | root@psmdb1:~# cat /etc/hosts 127.0.0.1 localhost 192.168.56.101 psmdb1 192.168.56.102 psmdb2 192.168.56.103 psmdb3 |
Choose a name for the replica set
We are now close to finalizing the configuration.
Now we have to choose a name for the replica set. We need to choose one and put t on each member’s configuration file. Let’s say we decide to use rs-test.
Put the replica set name into /etc/mongod.conf (the MongoDB configuration file) on each host. Enter the following:
1 2 | replication: replSetName: "rs-test" |
Restart the server:
1 | sudo service mongod restart |
Remember to do this on all the nodes.
That’s all we need to do to configure the replication at its most basic. There are obviously other configuration parameters we could set, but maybe we’ll talk about them in another post when discussing more advanced features. For this basic deployment we can assume that all the default values are good enough.
Initiate replication
Now we need to connect to one of the nodes. It doesn’t matter which, just choose one of them and launch mongo shell to connect to the local mongod instance.
Then issue the rs.initiate() command to let the replica set know what all the members are.
1 2 3 4 5 6 7 | mongo> rs.initiate( { ... _id: “rs-test”, ... members: [ ... { _id: 0, host: “psmdb1:27017” }, ... { _id: 1, host: “psmdb2:27017” }, ... { _id: 2, host: “psmdb3:27017” } ... ] }) |
After issuing the command, MongoDB initiates the replication process using the default configuration. A PRIMARY node is elected and all the documents will be created by now will be asynchronously replicated on the SECONDARY nodes.
We don’t need to do any more. The replica set is now working.
We can verify that the replication is working by taking a look at the mongo shell prompt. Once the replica set is up and running the prompt should be like this on the PRIMARY node:
1 | rs-test:PRIMARY> |
and like this on the SECONDARY nodes:
1 | rs-test:SECONDARY> |
MongoDB lets you know the replica role of the node that you are connected to.
A couple of useful commands
There are several commands to investigate and to do some administrative tasks on the replica set. Here are a couple of them.
To investigate the replica set configuration you can issue rs.conf() on any node
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 | rs-test:PRIMARY> rs.conf() { "_id" : "rs-test", "version" : 68835, "protocolVersion" : NumberLong(1), "members" : [ { "_id" : 0, "host" : "psmdb1:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 1, "host" : "psmdb2:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 }, { "_id" : 2, "host" : "psmdb3:27017", "arbiterOnly" : false, "buildIndexes" : true, "hidden" : false, "priority" : 1, "tags" : { }, "slaveDelay" : NumberLong(0), "votes" : 1 } ], "settings" : { "chainingAllowed" : true, "heartbeatIntervalMillis" : 2000, "heartbeatTimeoutSecs" : 10, "electionTimeoutMillis" : 10000, "catchUpTimeoutMillis" : 60000, "getLastErrorModes" : { }, "getLastErrorDefaults" : { "w" : 1, "wtimeout" : 0 }, "replicaSetId" : ObjectId("5aa2600d377adb63d28e7f0f") } } |
We can see information about the configured nodes, whether arbiter or hidden, the priority, and other details regarding the heartbeat process.
To investigate the replica set status you can issue rs.status() on any node
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | rs-test:SECONDARY> rs.status() { "set" : "rs-test", "date" : ISODate("2018-05-14T10:16:05.228Z"), "myState" : 2, "term" : NumberLong(47), "syncingTo" : "psmdb3:27017", "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : { "lastCommittedOpTime" : { "ts" : Timestamp(1526292954, 1), "t" : NumberLong(47) }, "appliedOpTime" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) }, "durableOpTime" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) } }, "members" : [ { "_id" : 0, "name" : "psmdb1:27017", "health" : 1, "state" : 2, <strong>"stateStr" : "SECONDARY"</strong>, "uptime" : 392, "optime" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) }, "optimeDate" : ISODate("2018-05-14T10:16:04Z"), "syncingTo" : "psmdb3:27017", "configVersion" : 68835, "self" : true }, { "_id" : 1, "name" : "psmdb2:27017", "health" : 1, "state" : 1, <strong>"stateStr" : "PRIMARY"</strong>, "uptime" : 379, "optime" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) }, "optimeDurable" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) }, "optimeDate" : ISODate("2018-05-14T10:16:04Z"), "optimeDurableDate" : ISODate("2018-05-14T10:16:04Z"), "lastHeartbeat" : ISODate("2018-05-14T10:16:04.832Z"), "lastHeartbeatRecv" : ISODate("2018-05-14T10:16:03.318Z"), "pingMs" : NumberLong(0), "electionTime" : Timestamp(1526292592, 1), "electionDate" : ISODate("2018-05-14T10:09:52Z"), "configVersion" : 68835 }, { "_id" : 2, "name" : "psmdb3:27017", "health" : 1, "state" : 2, <strong>"stateStr" : "SECONDARY"</strong>, "uptime" : 378, "optime" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) }, "optimeDurable" : { "ts" : Timestamp(1526292964, 1), "t" : NumberLong(47) }, "optimeDate" : ISODate("2018-05-14T10:16:04Z"), "optimeDurableDate" : ISODate("2018-05-14T10:16:04Z"), "lastHeartbeat" : ISODate("2018-05-14T10:16:04.832Z"), "lastHeartbeatRecv" : ISODate("2018-05-14T10:16:04.822Z"), "pingMs" : NumberLong(0), "syncingTo" : "psmdb2:27017", "configVersion" : 68835 } ], "ok" : 1 } |
Here we can see for example if nodes are reachable and are running, but in particular we can see the role they have at this moment: which is the PRIMARY and which are SECONDARY
Test replication
Finally, let’s try to test that the replication process is really working as expected.
Connect to the PRIMARY node and create a sample document:
1 2 3 4 5 6 7 8 9 10 11 12 | rs-test:PRIMARY> use test switched to db test rs-test:PRIMARY> db.foo.insert( {name:"Bruce", surname:"Dickinson"} ) WriteResult({ "nInserted" : 1 }) rs-test:PRIMARY> db.foo.find().pretty() { "_id" : ObjectId("5ae05ac27e6680071caf94b7") "name" : "Bruce" "surname" : "Dickinson" } |
Then connect to a SECONDARY node and look for the same document.
Remember that you can’t connect to the SECONDARY node to read the data. By default reads and writes are allowed only on the PRIMARY. So, if you want to read data on a SECONDARY node, you first need to issue the rs.slaveOK() command. If you don’t do this you will receive an error.
1 2 3 4 5 6 7 8 9 10 11 12 | rs-test:SECONDARY> rs.slaveOK() rs-test:SECONDARY> show collections local <strong>foo</strong> rs-test:SECONDARY> db.foo.find().pretty() { "_id" : ObjectId("5ae05ac27e6680071caf94b7") "name" : "Bruce" "surname" : "Dickinson" } |
As we can see, the SECONDARY node has replicated the creation of the collection foo and the inserted document.
This simple test demonstrates that the replication process is working as expected.
There are more sophisticated features to investigate the replica set, and for troubleshooting, but discussing them it’s not in the scope of this post.
In Part 3, we’ll show how to encrypt the internal replication process we have deployed so far.
Read the first post of this series: Deploy a MongoDB Replica Set with Transport Encryption
Hi,
thank you for this blog, I like how clearly it’s written. I am coming from the non-Percona MongoDB and one thing is not clear to me. You do not need Config Servers in Percona MongoDB?
Do I understand it right that you just start a repl. cluster of Percona mongod servers and after that start mongos instance and that’s it? Or I just normally run repl. cluster of config servers and also another repl. cluster. Are the config servers necessary?
Regards
Tomas
Hi Tomas and thanks for appreciation.
Config Servers are only needed in case you are using sharding. In such a case you need to deploy any single shard as a replica set and even the config servers as a group of at least 3 replica set nodes.
Even mongos is needed only if you have shards.
If you’re deploying a simple Replica Set without sharding, nor mongos nor config servers are needed.
Thank you. That cleared it for me.