At AWS Re:Invent 2018 there were many great announcements of AWS New Services and New Features, but one basic feature that I’ve been waiting for years to be released is still nowhere to be found.
AWS Elastic Block Storage (EBS) is great and it’s got better through the years, adding different storage types and features like Provisioned IOPS. However, it still has the most basic inconvenient requirement – I have to decide in advance how much space I need to allocate, and pay for all of that allocated space whether I use it or not.
It would be so much better if AWS would allow true consumption model pricing with EBS, where you pay for the storage used, not the storage allocated. This is already the case for S3, RDS, and even EC2 instances (with Unlimited Option on T2/T3 Instances), not to mention Serverless focused services.
For example, I would love to be able to create a 1TB EBS volume but only pay for 10GB of storage if I only use this amount of space.
Modern storage subsystems do a good job differentiating between the space available on the block device and what’s being used by user files and filesystem metadata. The space that’s not allocated any more can be TRIMmed. This is a basic requirement for working well on flash storage, and as modern EC2 instances already provision EBS storage as emulated NVMe devices, I would imagine Amazon could hook into such functionality to track space actually used.
For us at Percona this would make shipping applications on AWS Marketplace much more convenient. Right now for Percona Monitoring and Management (PMM) we have to choose how much space to allocate to the EBS volume by default, picking between it being expensive to run because of paying for the large unused EBS volume or setting a very limited by default capacity that needs user action to resize the EBS volume. Consumption-based EBS pricing would solve this dilemma.
This problem seems to be well recognized and understood. For example Pure Storage Cloud Block Storage (currently in Beta) is expected to have such a feature.
I hope with its insane customer focus AWS will add this feature in the future, but currently we have to get by without it.
—
Image: Arnold Reinhold [CC BY-SA 2.5], via Wikimedia Commons
This is an amazing feature request to have in AWS, lot of customers running their workload on io1 and gp2, will be beneficial if this truly turns out on the actual size of the disk consumed.
The one thing, which comes on top of my head is, it might relax our capacity planning efforts to some extent, on each cycle. Also a lot of cost-savings on AWS pay-as-you-go model, for Elastic Storages too.
+1
That would be amazing but I do not dream that big. I would be happy if at least they would get rid of the hard limit of not be able to make further storage modifications for six hours or while the DB instance status is storage-optimization that affects both EBS and RDS. It would not be truly elastic yet but it would be better than now.
Yeah. This is a challenge I think especially if you’re increasing storage capabilities and your change was not enough 🙂
You can look into EFS, which sounds like it more or less does what you want https://aws.amazon.com/efs/
It has been a couple of years, but when we implemented EFS when reading/writing a few large files, the performance was sufficient for us. When we implemented a system that had a lot of small reads/writes to files we experienced a lot of latency with the file access and reverted back to EBS.
In the database world we are not particularly fond running databases on the NFS and similar filesystems. I agree though for some workloads it can be possible but yet it does not offer the same flexibility as using the proven filesystem of choice on the block device