In this blog post, we will discuss how to deal with jumbo chunks in MongoDB.
Scenario: You are a MongoDB DBA, and your first task of the day is to remove a shard from your cluster. It sounds scary at first, but you know it is pretty easy. You can do it with a simple command:
1 | db.runCommand( { removeShard: "server1_set6" } ) |
MongoDB then does its magic. It finds the chunks and databases and balances them across all other servers. You can go to sleep without any worry.
The next morning when you wake up, you check the status of that particular shard and you find the process is stuck:
1 2 3 4 5 | "msg" : "draining ongoing", "state" : "ongoing", "remaining" : { "chunks" : NumberLong(3), "dbs" : NumberLong(0) |
There are three chunks that for some reason haven’t been migrated, so the removeShard command is stalled! Now, what do you do?
Find Chunks That Cannot Be Moved
We need to connect to mongos and check the catalog:
1 2 3 | mongos> use config switched to db config mongos> db.chunks.find({shard:"server1_set6"}) |
The output will show three chunks, with minimum and maximum _id keys, along with the namespace where they belong. But the last part of the output is what we really need to check:
1 2 3 4 5 6 7 8 9 10 11 | { [...] "min" : { "_id" : "17zx3j9i60180" }, "max" : { "_id" : "30td24p9sx9j0" }, "shard" : "server1_set6", "jumbo" : true } |
So, the chunk is marked as “jumbo.” We have found the reason the balancer cannot move the chunk!
Jumbo Chunks and How to Deal With Them
So, what is a “jumbo chunk”? It is a chunk whose size exceeds the maximum amount specified in the chunk size configuration parameter (which has a default value of 64 MB). When the value is greater than the limit, the balancer won’t move it.
That’s just the concept though. As a concrete implementation, it is a chunk that has been flagged as being jumbo, which will happen after a splitChunk command finds it cannot split a range of documents into segments smaller than the settings-defined chunk size. splitChunk commands are typically executed by the balancer’s moveChunk commands or the background auto-splitting process.
E.g. imagine a shard key index of {“surname”: 1, “given_name”: 1}. This is a non-unique tuple because humans, regrettably, do not have a primary key. If you have 100,000 documents for {“surname”: “Smith”, “given_name”: “John”, …} there is no opportunity to split them apart. The chunk will, therefore, be as big as those 100,000 documents.
How to Clear the “Jumbo” Flag
In 4.0.15+ use the clearJumboFlags command.
In older versions, you do it manually by removing that “jumbo” field from the documents in the config db that define chunk ranges the mongos nodes and shard nodes refer to.
1 2 3 4 | db.getSiblingDB("config").chunks.update( {"ns": <your_sharded_user_db.coll>, "jumbo": true}, {$unset: { "jumbo": "" }} ); |
Dealing With the Indivisible
Starting in MongoDB 4.4 you will be able to use the refineCollectionShardKey command to add another field to the shard key, as a suffix field. Eg. Change {“surname”: 1, “given_name”: 1} to {“surname”: 1, “given_name”: 1, “date_of_birth”: 1}
But if you have MongoDB version <= 4.2 a chunk can’t be moved if it exceeds the chunk size setting. In the can’t-drain problem scenario described in the top paragraph you will have to do one of the following things to finish draining the shard so you can run the final removeShard :
- Move the jumbo chunks after raising the chunk size setting
- Iterate all the jumbo chunks Use the dataSize command to find out what the largest size is.
- Change the chunksize setting to be larger than that.
- Clear the jumbo flag (see sub-section above)
- Start draining again and wait for it to move the big chunks. Use the sh.moveChunk() command if you want to see them happen sooner rather than later.
- Don’t forget to change the chunk size back after.
- Delete that data for a while. Reinsert a copy after the shard draining is complete.
- You’ll still need to clear the jumbo flag (see sub-section above) before the now-empty chunk will be ‘moved’ to another shard.
When Jumbo Chunks in MongoDB Have ‘Lost Weight’ Since They Were Flagged
The jumbo flag, once attached, won’t be removed automatically even if documents were deleted, or otherwise shrunk, so the chunk is within the (default) 64MB size. It’s up to you to manually clear the jumbo flag (see sub-section above) and try again.
In theory, you don’t need to do any splits manually, but if you want to hurry up and get confirmation that the chunks can be split into small enough sizes see how to with the sh.splitAt() command documentation.
Many organizations require Enterprise subscription of MongoDB for the support coverage and additional features that it provides. However, many are unaware that there is an open source alternative offering all the features and benefits of a MongoDB Enterprise subscription without the licensing fees and the vendor lock-in.
In our joint NessPRO webinar geared at the Israeli marketplace, we will cover the Percona approach and features that make Percona Server for MongoDB an enterprise-grade, license-free alternative to MongoDB Enterprise edition.
