Comments on: LVM read performance during snapshots

By: Eric FRANCKX

Eric FRANCKX — Mon, 21 Oct 2013 15:26:52 +0000

Hi,
if you use a Netapp storage (NFS) do you think it will not go faster ? like some seconds (snapshot on WAFL)

Regards,

Eric

By: Peter Zaitsev

Peter Zaitsev — Mon, 12 Aug 2013 18:55:13 +0000

Yves,

Got it. I do not recognize there are so many drives. On such large number of drive indeed read-ahead will be reading factor.
with 120MB/sec and 60 drives it is just 2MB/sec per drive which is very reachable even with semi-random IO

By: Yves Trudeau

Yves Trudeau — Mon, 12 Aug 2013 18:33:59 +0000

Peter, these are all spinning drives(15k rpm) , both for LVM and ZFS servers. The difference is really a matter of read-ahead, if I disable the caching with “zfs set primarycache=metadata …”, the read performance falls by a factor of 60x. There’re approximately 60 drives in the zfs pool.

By: Peter Zaitsev

Peter Zaitsev — Mon, 12 Aug 2013 17:44:39 +0000

Yves,

Is it same volume the – legacy drives not SSDs ?

Random updates could cause either sequential reads on the old data being much slower (snapshot) or on the currently active copy…. unless you can get some magic 🙂

By: Yves Trudeau

Yves Trudeau — Mon, 12 Aug 2013 17:39:53 +0000

Just ran a similar test on ZFS and I got about 120MB/s while reading over 400GB. This is quite good, especially when considering there’re 4 mysql instances running on that servers.

By: Yves Trudeau

Yves Trudeau — Mon, 29 Jul 2013 14:37:28 +0000

Sorry for my slow replies to the comments, I posted just before leaving for vacations. After some thought and discussion with Peter, the difference is the added random IO ops required to perform what would normally be a sequential read. I’ll do a comparison with ZFS today.

By: Peter Zaitsev

Peter Zaitsev — Tue, 09 Jul 2013 23:26:47 +0000

There might be some misconception about what is happening here. Typically when you create LVM snapshot initially the reads from snapshot (assuming no other disk operations) will be as fast as from the origin – both for sequential and random IO. However if you leave database operate when read performance will degrade, especially for _sequential_ reads where it can go down 10x or more. The main reason for that not scanning COW zone but rather the fact there are a lot of blocks in snapshot which have to be recovered from various COW locations – so what is say 1MB sequential read for original partition might end up being multiple non sequential IO operations.

By: Rolf

Rolf — Tue, 09 Jul 2013 19:21:32 +0000

Peter had a related writeup back in 2009 : http://www.mysqlperformanceblog.com/2009/02/05/disaster-lvm-performance-in-snapshot-mode/

By: Dennis Jacobfeuerborn

Dennis Jacobfeuerborn — Tue, 09 Jul 2013 17:56:21 +0000

Did you perform this benchmark with the old type snapshots or the new thinly provisioned ones? At least for writes the new implementation is dramatically faster than the old one as the COW is performed differently (and it has more features).

By: gebi

gebi — Tue, 09 Jul 2013 11:28:54 +0000

Yea, it would be _really_ interesting how ZFS compares to LVM snapshot based backup!

By: john

john — Tue, 09 Jul 2013 11:11:42 +0000

So, how does it compare to ZFS, which always uses COW?

By: Chris Boulton

Chris Boulton — Tue, 09 Jul 2013 11:09:13 +0000

LVM snapshot performance is something I’ve wanted to look at for a while, and at the same time do a study of performance of snapshots taken using Idera’s HotCopy (http://www.idera.com/productssolutions/freetools/sblinuxhotcopy).

The biggest benefit of HCP is that it doesn’t require free space in a volume group to store the changed deltas while a snapshot is active – it uses a file on the file system. The downside is of course, it uses a proprietary kernel module.