Allow parallel zstd compression when taking a base backup.
authorRobert Haas <rhaas@postgresql.org>
Wed, 30 Mar 2022 13:35:14 +0000 (09:35 -0400)
committerRobert Haas <rhaas@postgresql.org>
Wed, 30 Mar 2022 13:41:26 +0000 (09:41 -0400)
commit51c0d186d99a18e6aae53003f5138f20991e15a6
tree0c37c8899e6448c1bd56919741f58e8e5a99c7b5
parentc6863b85829149e2241faafa161b6c5af1f06cb9
Allow parallel zstd compression when taking a base backup.

libzstd allows transparent parallel compression just by setting
an option when creating the compression context, so permit that
for both client and server-side backup compression. To use this,
use something like pg_basebackup --compress WHERE-zstd:workers=N
where WHERE is "client" or "server" and N is an integer.

When compression is performed on the server side, this will spawn
threads inside the PostgreSQL backend. While there is almost no
PostgreSQL server code which is thread-safe, the threads here are used
internally by libzstd and touch only data structures controlled by
libzstd.

Patch by me, based in part on earlier work by Dipesh Pandit
and Jeevan Ladhe. Reviewed by Justin Pryzby.

Discussion: http://postgr.es/m/CA+Tgmobj6u-nWF-j=FemygUhobhryLxf9h-wJN7W-2rSsseHNA@mail.gmail.com
doc/src/sgml/protocol.sgml
doc/src/sgml/ref/pg_basebackup.sgml
src/backend/replication/basebackup_zstd.c
src/bin/pg_basebackup/bbstreamer_zstd.c
src/bin/pg_basebackup/t/010_pg_basebackup.pl
src/bin/pg_verifybackup/t/009_extract.pl
src/bin/pg_verifybackup/t/010_client_untar.pl
src/common/backup_compression.c
src/include/common/backup_compression.h