Comments on: Performance Evaluation of SST Data Transfer: Without Encryption (Part 1) https://www.percona.com/blog/performance-evaluation-of-sst-data-transfer-without-encryption-part-1/ Thu, 30 Mar 2017 17:50:13 +0000 hourly 1 https://wordpress.org/?v=6.5.2 By: Alexey Stroganov https://www.percona.com/blog/performance-evaluation-of-sst-data-transfer-without-encryption-part-1/#comment-10967943 Thu, 30 Mar 2017 17:50:13 +0000 https://www.percona.com/blog/?p=41697#comment-10967943 John,

there are few aspects here:

– with help of bbcp tool you can measure amount of time it will take in ‘ideal’ case to transferr 3TB of data in your environment. That tool allows to transfer data in parallel very efficiently:
time bbcp -P 2 -w 8M -s8 -N io ‘tar -c -O’ :’tar -C -xf -‘
– for socat you may try to adjust data transfer block size (-b) and rcvbuf/sndbuf

– ensure that storage/fs on joiner and donor sides tuned properly

– in general with current design xtrabackup may read/write/(de|en)crypt/(de)compress in parallel but streaming is serialized so even if you will make network streaming(socat or something else) in parallel xtrabackup still will output 1 stream only. However I would note that with new crc32 algo and “–parallel” option overhead from this serialization is quite low so difference in time for transferring of ‘pure’ data with bbcp and xtrabackup should be not so significant.

]]> By: john https://www.percona.com/blog/performance-evaluation-of-sst-data-transfer-without-encryption-part-1/#comment-10967941 Thu, 30 Mar 2017 15:14:33 +0000 https://www.percona.com/blog/?p=41697#comment-10967941 Thanks for the info. We were already using extra memory on the apply side and your suggestion seems to be worthwhile to help on the donor side, however, our bottleneck appears to be the socat transfer for a 3TB database. Any thoughts on how this can be made parallel as well?

]]>