For many this may be too much hassle for too little benefit, but for us it saves monthly transfer and lowers bandwidth use at the cost of a little CPU time.

We’ve encountered a bottle issue with gzip, as wal archives started to pile up in the pg_data/pg_wal folder.
To solve this, we’ve replaced it with “pigz -p 3 –fast” so it can use 3 cpu cores instead of one.

Best regards.

Multi-core replication/compression might be a good idea, as with TLS/zlib compression, one Xeon-D 2.7Ghz CPU core can only do ~50Mbit or 15-30MByte/sec; enough for us, but perhaps not for others.

Also, forgot to mention: for an initial base backup, the PostgreSQL environment is not included (as I guess is expected, as you’re not running the main binary or specifying a cluster). You seem to need:
OPENSSL_CONF=’/etc/postgresql/12/main/openssl.conf’ pg_receivewal … [with sslcompression=1 in the connection string, and the openssl configuration file above]

Thanks for posting!

It would be great to make a performance benchmark of various compression algorithms.

I think it makes sense to mention that gzip supports different methods of compression, see man gzip.

Also, it would be useful to check the zstd algorithm made by Facebook used as a compression algorithm for WAL.
https://facebook.github.io/zstd/

What do you think?

This means both savings on transfer and an increase in the amount of data that may be streamed (though since replication is done on a single core, CPU usage may be your limit, ~50Mbps – just right for making full use of DSL).

Unfortunately, it’s become increasing difficult to make this work on current operating systems, after padding-related vulnerabilities relating largely to website use of TLS gave compression within the protocol a bad name.

PostgreSQL hasn’t yet provided a solution for this, though, despite attempts to do so by some developers, so it’s still useful. Here’s what we do to get it to work.

First off, custom-compile openssl to re-enable SSL compression. You will need to redo this every time openssl/libssl1.1 updates (to remains secure and compressed).

On both SENDER and STANDBY:

apt-get source -t buster openssl
cd openssl[tab]

patch -p0 options |= SSL_OP_NO_COMPRESSION | SSL_OP_ENABLE_MIDDLEBOX_COMPAT;
+ ret->options |= SSL_OP_ENABLE_MIDDLEBOX_COMPAT;

ret->ext.status_type = TLSEXT_STATUSTYPE_nothing;

rm ssl/ssl_lib.c.orig

edit debian/rules
Change ‘no-zlib’ to ‘zlib-dynamic’
edit debian/changelog
Make an entire copy of the topmost segment and change the version to have ib at the end (don’t use a -ib as this breaks backported security fixes)
This will ensure it is viewed as the latest version and not updated.
CFLAGS=’-march=native’ DEB_BUILD_OPTIONS=’nocheck nodoc’ ionice -c3 nice dpkg-buildpackage -b -j4 –no-sign

sudo dpkg -i ../openssl_1.1.1d-0+deb10u1ib_amd64.deb ../libssl1.1_1.1.1d-0+deb10u1ib_amd64.deb ../libssl-dev_1.1.1d-0+deb10u1ib_amd64.deb
[adjust as appropriate]

Add to /etc/postgresql/12/main/environment
PGSSLCOMPRESSION = 1
OPENSSL_CONF = ‘/etc/postgresql/12/main/openssl.conf’

Create /etc/postgresql/12/main/openssl.cnf

openssl_conf = default_conf

[default_conf]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
MinProtocol = TLSv1.2
MaxProtocol = TLSv1.2
Compression = On

Note: Setting something like
CipherString = DEFAULT@SECLEVEL=2:+AES256:+CHACHA20
is not effective here. Ideally we’d use something like this on the sender to prefer AES-128 instead:
ssl_ciphers = ‘DEFAULT@SECLEVEL=2:+AES256:+CHACHA20’ # Prioritize better-performing (lower-CPU) AES-128 while exclude outdated ciphers
however doing so seems to preclude compression, probably because it is reset at the same time.

If requiring compression over a pgbouncer network connection, use:
server_tls_protocols = tlsv1.2
or, if the pgbouncer is local and the server remote
client_tls_protocols = tlsv1.2

You can test to see by setting log_connections = on and client_min_messages = notice on the server and checking to see whether compression=on is included within the log string. If it’s off, you’ve not done something right.