Since the release of Percona Server MongoDB 3.6.13 (PSMDB), you have been able to use Vault to store the encryption keys for data at rest encryption. Here’s how to set it up.
First, you need to have a Vault server up and running. My colleague, Jericho, has an article on setting up Vault for Percona Server titled Using the keyring_vault Plugin with Percona Server for MySQL 5.7. In this post, I will provide the same instructions for installing and setting up Hashicorp Vault for testing (Thank you Jericho!).
Vault Installation(run as root or use sudo):
1. Download, extract and install Vault:
1 2 3 | # wget https://releases.hashicorp.com/vault/1.3.2/vault_1.3.2_linux_amd64.zip # unzip vault_1.3.2_linux_amd64.zip # mv vault /usr/sbin |
Make sure to place it somewhere on your PATH
2. Place initial vault configuration in /etc/vault
1 2 3 4 5 6 7 8 9 10 11 12 | # mkdir /etc/vault # cd /etc/vault # cat vault.hcl listener "tcp" { address = "192.168.0.114:8200" tls_cert_file="/etc/vault/vault.crt" tls_key_file="/etc/vault/vault.key" } storage "file" { path = "/var/lib/vault" } disable_mlock=true |
You can use a port different than 8200. disable-mlock=true is needed if you want to start the Vault server as a non-root user.
3. Generate SSL certificates. To be able to create SSL certificates without going through prompts, we will place those entries in a configuration file:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | # cd /etc/vault # cat ssl.conf [req] distinguished_name = req_distinguished_name x509_extensions = v3_req prompt = no [req_distinguished_name] C = US ST = NC L = R O = Percona CN = * [v3_req] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer basicConstraints = CA:TRUE subjectAltName = @alt_names [alt_names] IP = 192.168.0.114 |
4. Run the commands below to generate the certificates and store them in /etc/vault.
1 2 3 | # cd /etc/vault # openssl req -config ssl.conf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout vault.key -out vault.crt # cat vault.key vault.crt > vault.pem |
5. Ensure that the keys are only accessible by the owner.
1 2 3 | # chmod 400 vault.key # chmod 400 vault.pem # chmod 400 vault.crt |
6. Set environment variables needed to access the vault.
1 2 | # export VAULT_CACERT=/etc/vault/vault.crt # export VAULT_ADDR='https://192.168.0.114:8200' |
7. Start Vault in the background.
1 2 | # vault server -config=/etc/vault/vault.hcl >> /var/log/vault.log 2>&1 & [1] 7336 |
8. Initialize Vault and store the unseal keys and the initial root token generated in this step:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | # vault operator init Unseal Key 1: Gdu0HtSSctKUvn0ssi+GWhKEDZBWMAmulKfiAroCt+iw Unseal Key 2: UEf61dpqPvjD+ftFaCJTy+KzKlSTIsT75qwe4gVLR9w4 Unseal Key 3: am/vq6P/SNs7aIKiLr7gEkeV8Pn/ilkV5r+HbwTpn7Vk Unseal Key 4: aZYbiu2C3zrDklyWxf0JaqAouGRM95g0a9vz1JIk6jHD Unseal Key 5: IApVQeN6kxCU1dr/H/Dc+1Afn6CJ5VVhGdAPNC1tGgVL Initial Root Token: s.AqVdpn2C4NVJgtkZkRrKeFtm Vault initialized with 5 key shares and a key threshold of 3. Please securely distribute the key shares printed above. When the Vault is re-sealed, restarted, or stopped, you must supply at least 3 of these keys to unseal it before it can start servicing requests. Vault does not store the generated master key. Without at least 3 key to reconstruct the master key, Vault will remain permanently sealed! It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information. |
9. Unseal the Vault by supplying the three unseal keys generated above. You need to unseal Vault every time the Vault server is started.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | # vault operator unseal Gdu0HtSSctKUvn0ssi+GWhKEDZBWMAmulKfiAroCt+iw Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 1/3 Unseal Nonce 6be37706-61cb-b372-cc88-b0e89e651aff Version 1.3.2 HA Enabled false # vault operator unseal UEf61dpqPvjD+ftFaCJTy+KzKlSTIsT75qwe4gVLR9w4 Key Value --- ----- Seal Type shamir Initialized true Sealed true Total Shares 5 Threshold 3 Unseal Progress 2/3 Unseal Nonce 6be37706-61cb-b372-cc88-b0e89e651aff Version 1.3.2 HA Enabled false # vault operator unseal am/vq6P/SNs7aIKiLr7gEkeV8Pn/ilkV5r+HbwTpn7Vk Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 5 Threshold 3 Version 1.3.2 Cluster Name vault-cluster-752be70f Cluster ID 32ebb89d-2583-23b6-513d-c44220e12ac0 HA Enabled false |
10. Now it’s time to log in to Vault and use the initial root token.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # vault login s.AqVdpn2C4NVJgtkZkRrKeFtm Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token s.AqVdpn2C4NVJgtkZkRrKeFtm token_accessor EZo7M0JbZjicINWMLee573qL token_duration ∞ token_renewable false token_policies ["root"] identity_policies [] policies ["root"] |
11. At this point, on Vault, you will need to enable the KV Secrets Engine – Version 2, the only engine supported by PSMDB. To do so, you will need to run the following:
1 2 | # vault secrets enable -path secret kv-v2 Success! Enabled the kv-v2 secrets engine at: secret/ |
12. Next, you will need to create a Vault Policy to access this datastore. As per https://www.vaultproject.io/docs/secrets/kv/kv-v2/, reading and writing versions need to be prefixed with the path data/.
1 2 3 4 | # cat mongodb.hcl path "secret/data/dc/*" { capabilities = ["create","read","update","delete"] } |
13. To upload the policy to the Vault server, run:
1 2 | # vault policy write mongodb-policy mongodb.hcl Success! Uploaded policy: mongodb-policy |
14. Finally, you will need to create an access token to be used by MongoDB. Remember to create a token per MongoDB instance.
1 2 3 4 5 6 7 8 9 10 | # vault token create -policy=mongodb-policy Key Value --- ----- token s.cFy5NxA72Wk7VhVH45VJ4Rib token_accessor yoVBWgUVtDpzIomIRGTvBRd6 token_duration 768h token_renewable true token_policies ["default" "mongodb-policy"] identity_policies [] policies ["default" "mongodb-policy"] |
You will need to use the token above to connect MongoDB to Vault.
15. Done.
Incorporating Vault in MongoDB
In MongoDB, you’ll need to create a token based on the generated token and CA file generated in the Vault. Let’s place them in /etc/mongodb:
1. Create a mongodb config directory that will store both token and CA file.
1 2 | # mkdir -p /etc/mongodb # chown mongod:mongod /etc/mongodb |
2. Place the token value in the token file and copy the contents of vault.crt from the Vault server.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # cd /etc/mongodb # cat token s.cFy5NxA72Wk7VhVH45VJ4Rib # cat vault.crt -----BEGIN CERTIFICATE----- MIIDbDCCAlSgAwIBAgIJAOsptGVHJfKYMA0GCSqGSIb3DQEBCwUAMEQxCzAJBgNV BAYTAlVTMQswCQYDVQQIDAJOQzEKMAgGA1UEBwwBUjEQMA4GA1UECgwHUGVyY29u YTEKMAgGA1UEAwwBKjAeFw0yMDAzMzAxODE4NDlaFw0zMDAzMjgxODE4NDlaMEQx CzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEKMAgGA1UEBwwBUjEQMA4GA1UECgwH UGVyY29uYTEKMAgGA1UEAwwBKjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC ggEBAKSRMI4/rbiWitDZtnDzlCVi0BDalmn4ieG93mt1Vn0IfN4NOmQeZ5fVRBve Y26ugAT7Nq92puvV6vLk3ky8K8dkNPLe4wPEMPWnrHJIS4zqcmJWbw618yymVufm h59cI/jyYCPMyLXkcZLuOnSZszoKnN/jEZq6WKNxgFuc6evzfxN4AbM7ffRk2mj5 FfMbtpToBgJS/U9UMS7EvzXYslUyq4bgBx3ygcmUyas7Ej7aie2fb0tzW/dMJ6/F M2kYnHTM+pxr+Eq5QndBcjgptv/iqdnmIbNHogY+4TOCQwyke83SphgIwtI9MCLN xUDN5nZEeRY8RINk3+irL/Kq0ocCAwEAAaNhMF8wHQYDVR0OBBYEFFS+R00NplQg c6XFyYNLmrMJFa4lMB8GA1UdIwQYMBaAFFS+R00NplQgc6XFyYNLmrMJFa4lMAwG A1UdEwQFMAMBAf8wDwYDVR0RBAgwBocEwKgAcjANBgkqhkiG9w0BAQsFAAOCAQEA h9dKQoTILcv43mVbowCENm0ujPCW74CM/VXc+k8s3A5+h2tHtrO6NYATQBA07L+k dwuethgqaqzbaEJqF766UiKv00hCkY8fMSyptLRtYG3WtxDWN/kGGvSjkzbYYqDP RK1X5eI7PT7rtyu+FMWEiFLgQE6e5pz8MOITvdtQviJ35jUD1FfCpzTWppqeCc0g 9dVhzej1KDYTpoTcUkGLYDF1AMwEdXXqI77zdbhzQfmSlt1hAJGBKJFvAGTM2XHb vd0QhgGbfzryv1b+eDTL7wWsYckNEcADpizHinxBpKC9/cnJZZtt6D35NvVAMkjy +8wuOopSb6PCjrxVAz75FA== -----END CERTIFICATE----- |
3. Ensure the files are readable by mongod only, as MongoDB will complain and fail during startup with lax permissions.
1 2 3 4 | # chown mongod:mongod /etc/mongodb/token # chown mongod:mongod /etc/mongodb/vault.crt # chmod 400 /etc/mongodb/token # chmod 400 /etc/mongodb/vault.crt |
4. Place in /etc/mongod.conf the Vault configuration under the security section. As a sample naming convention, given that the hostname of this server is psmongodb1, the secret path will be secret/data/dc/psmongodb1 as well:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 | # cat /etc/mongod.conf # mongod.conf, Percona Server for MongoDB # for documentation of all options, see: # http://docs.mongodb.org/manual/reference/configuration-options/ *** redacted *** processManagement: fork: true pidFilePath: /var/run/mongod.pid # network interfaces net: port: 27017 bindIp: 127.0.0.1 #security: security: enableEncryption: true vault: serverName: 192.168.0.114 port: 8200 secret: secret/data/dc/psmongodb1 tokenFile: /etc/mongodb/token serverCAFile: /etc/mongodb/vault.crt *** redacted *** |
5. Then start MongoDB:
1 | # systemctl start mongod |
You can only enable encryption on an empty database. It’s not possible to re-encrypt an existing, previously unencrypted database. One way to do it is to create a backup, stop mongod, delete data files, enable encryption, start mongod, and restore the database from backup. If you have a replica set, you can apply changes in a rolling manner (one member at the time). Data will be synced automatically from other nodes. Each node has to be encrypted separately.
6. Check the logs if using the key was successful:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | 2020-03-30T19:05:48.929+0000 I ACCESS [main] Initialized External Auth Session 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] MongoDB starting : pid=8026 port=27017 dbpath=/var/lib/mongo 64-bit host=psmongodb1.example.com 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] db version v3.6.17-4.0 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] git version: 96e9c7218eebc2995dd847d6185f20f102a86055 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] allocator: tcmalloc 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] modules: none 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] build environment: 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] distarch: x86_64 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] target_arch: x86_64 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongod.pid" }, security: { enableEncryption: true, vault: { disableTLSForTesting: false, port: 8200, secret: "secret/data/dc/psmongodb1", serverCAFile: "/etc/mongodb/vault.crt", serverName: "192.168.0.114", tokenFile: "/etc/mongodb/token" } }, storage: { dbPath: "/var/lib/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongo/mongod.log" } } 2020-03-30T19:05:49.024+0000 I STORAGE [initandlisten] Master key is absent in the Vault. Generating and writing one. 2020-03-30T19:05:49.076+0000 I STORAGE [initandlisten] Initializing KeyDB with wiredtiger_open config: create,config_base=false,extensions=[local=(entry=percona_encryption_extension_init,early_load=true,config=(cipher=AES256-CBC,rotation=false))],encryption=(name=percona,keyid=""),log=(enabled,file_max=5MB),transaction_sync=(enabled=true,method=fsync), 2020-03-30T19:05:49.556+0000 I STORAGE [initandlisten] Encryption keys DB is initialized successfully 2020-03-30T19:05:49.556+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=256M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),encryption=(name=percona,keyid="/default"),extensions=[local=(entry=percona_encryption_extension_init,early_load=true,config=(cipher=AES256-CBC)),local=(entry=percona_encryption_extension_init,early_load=true,config=(cipher=AES256-CBC)),], |
As per the logs, it was able to generate a master key and use the Percona encryption. At this point, data at rest encryption has been configured on this server.
7. Done.
Take note that if the Vault server is down, you will not be able to start up your server. Given this new dependency on obtaining the master key from Vault, be sure to design your architecture for fault tolerance.
Encryption Key Rotation
Each encryption key can be used only for a limited time. Rotating master key re-encrypts the keystore using a new master key. The newly generated master key is then stored in the Vault. The entire dataset isn’t re-encrypted.
1. Place in /etc/mongod.conf an extra line in the security section:
1 2 3 4 5 6 7 8 9 10 | # cat /etc/mongod.conf security: enableEncryption: true vault: serverName: 192.168.0.114 port: 8200 secret: secret/data/dc/psmongodb1 tokenFile: /etc/mongodb/token serverCAFile: /etc/mongodb/vault.crt rotateMasterKey: true |
2. Then restart MongoDB:
1 | # systemctl restart mongod |
You’ll get an error message (but that’s expected, as mongod process didn’t start successfully). You can check logs if the encryption key was successfully rotated:
1 2 3 4 5 6 7 | # tail /var/log/mongo/mongod.log 2020-03-30T19:05:48.929+0000 I STORAGE [initandlisten] Encryption keys DB is initialized successfully 2020-03-30T19:05:48.933+0000 I STORAGE [initandlisten] exception in initAndListend std::exception: master key rotation finished successfully, terminating 2020-03-30T19:05:48.933+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets... 2020-03-30T19:05:48.933+0000 I - [initandlisten] Stopping further Flow Control ticket acquisitions. 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] now exiting 2020-03-30T19:05:48.933+0000 I CONTROL [initandlisten] shutting down with code:100 |
3. Remove the extra line in /etc/mongod.conf configuration rotateMasterKey: true
4. Done. You can start mongod again, your key was rotated.
Please be advised that this is a simple guideline on how to set up a basic Vault server and should not be used as a template for production usage. We recommend using Percona Consulting Services to assist you in that matter.
Fatal Assertion 50944 at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 1073 i am getting this one
Thanks Jaime and Michal for your informative article. I do have a comment regarding the Vault token creation however – if you use the above commands, you will end up with a token that will expire in 768h that cannot be renewed.
I suggest something like:
# vault token create -period=768h -renewable -policy=mongodb-policy
You will then be able to periodically renew the token (so long as it hasn’t expired) by a curl command like:
# curl -s –header “X-Vault-Token: s.cFy5NxA72Wk7VhVH45VJ4Rib” –request POST https://192.168.0.114:8200/v1/auth/token/renew-self
You could alternatively renew using the accessor token.
Thank you, Craig for valuable input! Our intention was to keep it as simple as possible – to provide the setup for testing this feature. There are probably also many other points worth considering when configuring it for production workload.