A few weeks ago I sat down and talked about tuning MongoDB memory settings with Mike Grayson, one of our MongoDB Database Engineers here at Percona. If you have not seen the video I would recommend it. Since then I have heard from people in the community about other topics they would like to see. I was able to do a written interview with Mike today (with some great extra details from Vinodh), asking them to provide us a MongoDB Replication and Sharding 101 overview based on the questions I heard from the community. Thank you, Mike and Vinodh, for writing up answers to our questions! Also, stay on the lookout for a video chat I did with our MongoDB Software product owner Akira, asking him to explain sharding and replication at a high level.
In MongoDB what is the difference between sharding and replication?
Mike Grayson: Sharding is the act of partitioning your collections so that parts of your data are dispersed among multiple servers called shards. Replication, or Replica Sets in MongoDB parlance, is how MongoDB achieves high availability, Replica Sets are a Primary, and 0 to n amount of secondaries which have read-only copies of the data and possibly an arbiter, which is a non-data bearing vote only member. Upon Primary failures, one of the available secondaries will be elected as the new Primary.
Vinodh Krishnaswamy added: Don’t forget sharded clusters have three components – shards (where original data resides), config servers (holds information about data like which data resides which shard), and mongos (a router – the application connects to the cluster via mongos). For high availability, you would need to implement both the shards and the config servers as replica sets.
When should you start to look at MongoDB sharding?
Mike Grayson: Typically you should look at sharding when either your working set has outpaced the available resources on your machine (think vCPU or RAM) and scaling up either isn’t possible or isn’t feasible, perhaps due to cost OR when your data set is growing too large (Some people say that you should start looking at around 1-2TB, but this maybe too large and you should start thinking about sharding strategies in the 200GB-500GB range).
Vinodh Krishnaswamy’s tip: Think early on the growth and potential size of your databases and plan out your sharding strategy.
Can you shard MongoDB without replicating? When would you do this?
Mike Grayson: Can you? Yes. Should you? No. Sharding without replicating doesn’t enable the oplog which means backups aren’t possible for your sharded cluster. Why might someone do this? Think log data that’s easily reproduced or other data that’s not that important such as transient data. Generally speaking, those reasons are few and far between, and setting up a replica set is still a best practice in those situations.
Are there downsides to sharding MongoDB too early?
Mike Grayson: Cost, the amount of infrastructure and resources needed to support a bare-bones production level sharded cluster is at least twice that of a 3 node replica set. However, if you know your data set is going to grow, spinning up a sharded cluster of 1 shard puts you in a great spot to scale out quickly and easily.
Vinodh Krishnaswamy added: One another drawback of sharding too late however is the extra potential load when balancing. Migrating the data chunks from old shards to the new ones is an expensive operation. When you have TBs of data, a lot of chunks will need to be migrated and might affect the performance and stability of your regular workload. To mitigate this you will need to set a window to balance the shards and to allow chunk migrations only to occur at a certain period of the day!
Are there a minimum number of MongoDB nodes you need to shard?
Mike Grayson: 2 is the absolute minimum, 1 node with a mongos (query router used to route queries to the correct shard)/config server and another node for the shard. This would be able to support a 1 shard cluster. For production quality, you’d want a minimum of 7 nodes, 1 mongos on its own, 3 config servers and a 3 shard replica set nodes for a 1 shard cluster.
If you have a small 3 node MongoDB setup can you have both sharding and replication?
Mike Grayson: Yes, if it’s not expected to support production load you could have 3 nodes that support a sharded cluster that utilizes replica sets. They would be replica sets of 1 member though OR you could stack multiple mongod nodes on a single host, which is generally not a good idea in Production, but possible in test/dev type environments
Vinodh Krishnaswamy added: I did a webinar a little over a year ago where I showed and walked through how to design your sharding topology. If people would like a deeper dive I would encourage readers to check it out:
Recorded Webinar: MongoDB Sharded Cluster & HA – How to Design your Topology
When happens if you get the MongoDB shard key wrong?
Mike Grayson: Getting the shard key wrong can lead to unbalanced data across shards or “hot” shards where all new data goes to the same shard. In MongoDB 4.4, they’ve added refinable shard keys which makes this slightly less painful than previous as you can add (but not remove) fields from your shard key, but choosing your shard key should still be something you plan and test extensively before moving to production.
What is the most common issue people run into when sharding MongoDB?
Mike Grayson: Poor Shard Keys leading to data imbalance, whether it be through jumbo chunks (a chunk is a piece of your data that MongoDB can move around from shard to shard, jumbo chunks are too big to move and get stuck on a shard without intervention), hot shards where new data is all being written to one shard and the other shards struggle to keep up.
Vinodh Krishnaswamy: Not using the shard key fields in their query which will lead to broadcasting the requests to all shard for the data and then merging the data together to provide the result. This is a much more expensive operation than when you query with the shard key. With queries including shard key fields, the requests are redirected to only the shards where the requested data resides
Thank you, Mike and Vinodh for taking the time to write up answers to these questions!
