Then the double write can be disabled without risking data corruption?

I’ve seen similarly impressive results mapping ZFS Journaling Device to our Flashtec NVRAM Drive (check it out here – pmcs.com/products/storage/flashtec_nvram_drives). Pretty much doubles your throughput in write intensive workloads.

Contact me if you would like to try this with the fastest NVRAM Cache Card in the market!

Cheers,

Fabian

It is true that, when using data=journal, EXT4 will guarantee that application data are double-written to both the journal and the main filesystem. However _if the application crashes during a write_ the filesystem will receive incomplete data, yet it does not know anything about that.

Example: a C program is writing a large enough (eg: 128KB) buffer using a write call. Suddenly, it is killed just when the write call is transferring the application buffer’s content into OS pagecache. After some time (or by issuing a sync) the filesystem receive the partial data and double-write them. The key point here is that, while data are consistent from filesystem view, they are _not_ consistent from the application view.

Using only small buffer (eg: 16KB) will rarely trigger the problem, but using bigger buffer surely will (the example with 128 KB buffer is quite telling).

So: do NOT disable InnoDB doublewrite, unless you _really_ know not only what you are doing, but the OS-level implication in doing that.

My colleague Sunny Bains pointed out that true copy-on-write file systems could probably guarantee atomic write operations. I do not see how an update-in-place file system like ext4 could guarantee it.

As you see, it is always a multiple of 16KB. Could you post your whole code with the “

” tags

Actually, I ran a similar test on Monday when my colleague pointed out that I am missing O_DSYNC.

More specifically, I used
int fd = open(“testfile”, O_WRONLY | O_CREAT | O_TRUNC | O_DSYNC, 0666);
in my test program.

Some commands that I used:

/sbin/mkfs.ext4 /dev/shm/u
sudo mount /dev/shm/u /mnt
sudo chmod 777 /mnt
cd /mnt
/tmp/a.out & sleep 0.00001; kill %1
wc -c testfile
cd
umount /mnt
sudo mount /dev/shm/u /mnt -t ext4 -o data=journal
cd /mnt
/tmp/a.out & sleep 0.00001; kill %1
wc -c testfile

After the remounting, the file was smaller (I got 16384, 36864, 98304, and 45056 bytes on 4 attempts). With the defaults (no data=journal), the size with my only attempt was 352256 bytes. So, the data=journal is definitely slowing it down, but not making the write() atomic. The mount options according to /proc/mounts are as follows:

/dev/loop0 /mnt ext4 rw,relatime,nodelalloc,data=journal 0 0

The file /proc/version identifies the kernel as follows:

Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version
4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23)

Also, “man 2 open” does not promise to me that O_DSYNC or the stronger O_SYNC would make writes atomic. It only seems to promise that if the write completes, it will be completely recovered from the journal, should the file system crash. In this case, the write would not complete on my system, because it was interrupted by a signal.

If this killed write happened to an InnoDB data file with the doublewrite buffer disabled, the data file would not be recoverable when mysqld is restarted. If the kill was soon enough followed by a full system crash (or pulling the plug), then you might be safe, because the partial write might not have been written to the file system journal yet.

Can you please retry with a bigger write or a smaller sleep delay? I think that it is easier to interrupt a write with a signal when the request is bigger than the 16384 bytes that you were using. Note that this is not only an academic request: I seem to remember that InnoDB can combine writes of adjacent pages into a single request.

Try the following code:

#define _GNU_SOURCE

#include
#include
#include

int main ()
{
FILE *fp;
int fd;
char *str;

str = (char *) malloc(16384);
memset(str,0,16384);

fd = open( “/var/lib/mysql/append16kb.txt” , O_DSYNC | O_WRONLY | O_TRUNC);

while (1) {
write(fd, str, 16384);
fdatasync(fd);
}

close(fd);

return(0);
}

And then, pull the plug on the server or kill -9 the process. I never got only part of a 16KB chunk. If you remove the O_DSYNC, you got 75% of chance of getting a partial write.

So it seems ZFS is OK.

char buf[10485760];

int main (int argc, char**argv)
{
write(1, buf, sizeof buf);
return 0;
}

Invoke this as
./a.out > testfile & sleep 0.0001; kill %1
[hit enter to see that bash reports the job as terminated]
wc -c testfile

All data is committed into the journal prior to being written into the main filesystem.

There is no mention of what happens when a write is interrupted by a signal. Are you trying to say that there is some ext4 file system mode where writes do not take effect until an explicit fsync() is submitted? And where non-fsync()ed writes would be rolled back if the process was killed?

The case that I am talking about is that the process is killed during a write operation. Because I do not remember if data=journal was explicitly used on the system where the problem occurred during our internal QA, I repeated my test with the follownig program, on an ext4 file system mounted in data=journal mode:

cat > t.c << EOF
#include

char buf[10 < /some/where/test/file& sleep 0.0001; kill %1
ls -l /some/where/test/file

Every time I run this, the file size will be a multiple of 4096 bytes, not the full 10MiB that was submitted.
OK, InnoDB does not use the write(2) system call but pwrite(2) or aio, but the semantics should be similar.

My first experiment was on /dev/shm (tmpfs). I repeated it also with an ext4 file system like this:

dd if=/dev/zero of=/dev/shm/t bs=1M count=50
/sbin/mke2fs /dev/shm/t
sudo mount /dev/shm/t /mnt -t ext4 -o data=journal
sudo chmod 777 /mnt
./a.out > /mnt/t& sleep 0.0001; kill %1
wc -c /mnt/t
98304 /mnt/t
./a.out > /mnt/t& sleep 0.0001; kill %1
wc -c /mnt/t
118784 /mnt/t

Again, not all 10MiB were written. Instead 24 or 29 blocks of 4KiB were written. The latter count is not a multiple of 16KiB (the default innodb_page_size).

Did you successfully test disabling the InnoDB doublewrite buffer and then repeatedly killing and restarting the server during a write-heavy workload, while using some other size than innodb_page_size=4k? As far as I can tell, this can only be guaranteed work on a file system where writes are supposed to be atomic, such as FusionIO NVMFS.

Our QA engineer only disables the doublewrite buffer when using the 4K page size on Linux, to avoid unnecessary noise.

Note: InnoDB crash recovery always requires the file pages to be readable. We could be more robust and skip the prior reading of the page if the redo log contains MLOG_INIT_FILE_PAGE or similar.

The granularity of the write operation is enforced to 16KB by the InnoDB fsyncs that happens at multiple of the page size provided that the file is opened use O_DSYNC (or O_SYNC actually). If you watch the server metrics, you’ll see that the data is still written twice, once to ext4 journal and once to the ibd file. I invite you to test. I never corrupted my dataset even when pulling the power plug in the middle of a heavy write benchmark and I did it often… After a crash, the filesystem will use its transactional journal to fully write an InnoDB page that would have been partially written.

The partial page writes in data files are independent of the redo log flushing. No matter which setting you use for the redo log flushing, disabling the doublewrite buffer is dangerous unless the innodb_page_size matches the granularity of the page writes. The doublewrite buffer only covers writes to the data files.

Partial writes are not that much of a problem in the redo log, because the redo log block size is smaller, and crash recovery would ignore the corrupted partially written last redo log blocks, replaying the log up to the last seen LSN.

Thanks!

Tim