Reclaim Disk Space in Percona Server for MongoDBCompaction in Percona Server for MongoDB (PSMDB)

There are two ways to reclaim disk space in Percona Server for MongoDB (PSMDB): run compact on nodes or resync the node. In this blog, we will see the best practice for reclaiming the fragmented space on a disk in PSMDB using compact.

Disk storage is a critical resource for any scalable database system. As the database grows and multiple write operations run, this contiguous space gets fragmented into smaller blocks with chunks of free space in between. The typical and fastest solution is to increase the disk size. However, if the application team started to delete data regularly and optimize algorithms to reduce redundant data, there are ways that can help you regain the free space without having to scale your disk size. Simply deleting documents cannot reduce PSMDB disk usage, but if you drop the collection it will delete the data file and immediately release the space, in most cases dropping the collection is not possible in production environments.

By reducing the disk space, it will not only save storage space but also will save money. We have done this for one of our customers whose environment was running on the cloud. The customer was using 4TB of disk space on each node of eight shards, i.e. 96TB in total for all shards. We archived the data, freed up space, and ran Compact to reclaim the disk space. After that, we were able to reduce the disk footprint from 4TB to 2.5TB on each node of all the shards i.e. from 96TB to 60TB. It significantly reduced the customer cloud bill.

What compact does

The WiredTiger storage engine maintains lists of empty records in data files as it deletes documents. This space can be reused by WiredTiger, but will not be returned to the operating system.

However, if you delete a lot of documents and want the WiredTiger storage engine to release this empty space to the operating system, you can defragment your data file. This is possible using the compact command.

Let’s see how data is stored and what happens when we delete the data and follow that run the compact.

  1. When we are inserting the data, new blocks are allocated to the data file, and the size keeps increasing as soon as new data is inserted:

       2. Now some data has been deleted/removed from the collection:

In step two, you can see when documents are removed or deleted and how space is not released.

       3. When you will perform the compact operation:

You can see above how the space is released and defragmented after the compact operation.

How do you check the storage stats?

PSMDB has a command db.stats() that can provide insights into the storage statistics of databases.

 

dataSize:
The total size in bytes of the uncompressed data held in this database.

storageSize:
The total amount of disk space allocated to all collections in the database.

fsUsedSize:
The total size of all disk space in use on the filesystem where PSMDB has the data directory.

fsTotalSize:
The total size of all disk capacity on the filesystem where PSMDB has the data directory.

You can see a difference between the storageSize and dataSize above because in WiredTiger, storageSize may be smaller than dataSize as compression is enabled by default. 

When a large chunk of data is deleted from a collection and the collection never uses the deleted space for new documents, this space needs to be returned to the operating system so that it can be used by your other databases or collections. 

Use the below command to check how much data you can reclaim in a collection:

Note: The above command output will come in bytes.

Use the below command to check how much data you can reclaim per collection of the database:

Prerequisites

Once you have the details using the above command and decided which collection you want to compact and reclaim space, please keep in mind the below things before running the compact command:

  1. Always take the full backup of the database.
  2. The user should have the required privilege to run the compact command.
  3. Check if you have enough replication oplog window.
  4. Always run compact on the secondary nodes/hidden nodes or nodes having low priority following with the Primary node in last after stepping it down.
  5. In a replica set, compact command need to be run on each node.
  6. In a shard cluster, a compact command need to be run on each node of every shard. Compact cannot be run against the mongos. 
  7. When you try to compact a collection, a write lock will be added to the database where the collection is located, which may cause all read/write requests to the database to be blocked.
  8. If you are running PSMDB 4.4 or a newer version, compact will not block CRUD operations as it did in earlier versions.
  9. In PSMDB 4.4 or newer, compact will only block the metadata operations like dropping a collection, dropping the index, and creating a new index.

Use the below command to compact the single collection of the database:

Use the below command to compact all collections of the database:

You can check the progress of compaction in mongo logs or by running db.currentOp() command in another shell instance.

Once the collections are compacted, please check the reclaimed space using the same command which we used to check how much space we can reclaim. Also, you can check the disk space on the OS level as well.

Conclusion

Sometimes when a large collection is compacted, the compact command immediately returns OK, but in reality, the physical space of the collection remains unchanged. This is because WiredTiger deems that the collection does not need to be compacted. In order to overcome this, you need to run the compact command again till it releases the space.

Before PSMDB 4.4, it was always advisable to run compact in a scheduled maintenance window due to the nature of the command which blocks all the read/write operations. Starting from the PSMDB 4.4, you can plan it to run at any time.

Percona Server for MongoDB is an open-source replacement for MongoDB Community Edition that combines all of the features and benefits of MongoDB Community Edition with enterprise-class features developed by Percona: LDAP Authentication and Authorization, Audit Logging, Kerberos Authentication, and hot backups

To learn more about the enterprise-grade features available in the vendor lock-in free Percona Server for MongoDB, we recommend going through our blog MongoDB: Why Pay for Enterprise When Open Source Has You Covered? 

Subscribe
Notify of
guest

3 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Dharmendra

I would like to appreciate the blog post written in a very compactful manner to explain the compact behaviour.

It would have been better if you have mentioned how much time it took to successfully perform compact on this 96TB of data?

Also have you seen any replication lag between the nodes in specific shard?

1
Reply
hcymysql

m5_rs1:PRIMARY> db.serverStatus().version
5.0.23-20
m5_rs1:PRIMARY> db.version()
5.0.23-20
m5_rs1:PRIMARY> db.runCommand({compact: “t1” })
{
“ok” : 0,
“errmsg” : “will not run compact on an active replica set primary as this
 is a slow blocking operation. use force:true to force”, “$clusterTime” : {
“clusterTime” : Timestamp(1708440622, 1),
“signature” : {
“hash” : BinData(0,”SHVL8Gc3Cic+mKsY65WnE/k8YDg=”),
“keyId” : NumberLong(“7327091310781792261”)
}
},
“operationTime” : Timestamp(1708440622, 1)
}

0
Reply
hcymysql

m5_rs1:PRIMARY> db.serverStatus().version
5.0.23-20
m5_rs1:PRIMARY> db.version()
5.0.23-20
m5_rs1:PRIMARY> db.runCommand({compact: “t1” })
{
“ok” : 0,
“errmsg” : “will not run compact on an active replica set primary as this
 is a slow blocking operation. use force:true to force”, “$clusterTime” : {
“clusterTime” : Timestamp(1708440622, 1),
“signature” : {
“hash” : BinData(0,”SHVL8Gc3Cic+mKsY65WnE/k8YDg=”),
“keyId” : NumberLong(“7327091310781792261”)
}
},
“operationTime” : Timestamp(1708440622, 1)
}

0
Reply