Add a new WAL summarizer process.
authorRobert Haas <rhaas@postgresql.org>
Wed, 25 Oct 2023 16:57:22 +0000 (12:57 -0400)
committerRobert Haas <rhaas@postgresql.org>
Thu, 30 Nov 2023 13:43:52 +0000 (08:43 -0500)
commitab3002d1908634f24775af799c0e6c822d598441
tree9a0346b47893461b53d906c13fd02ffd5dbdb3c4
parent9c31fb418c22bb02be5bc0e3b76fb6dca58cf252
Add a new WAL summarizer process.

When active, this process writes WAL summary files to
$PGDATA/pg_wal/summaries. Each summary file contains information for a
certain range of LSNs on a certain TLI. For each relation, it stores a
"limit block" which is 0 if a relation is created or destroyed within
a certain range of WAL records, or otherwise the shortest length to
which the relation was truncated during that range of WAL records, or
otherwise InvalidBlockNumber. In addition, it stores a list of blocks
which have been modified during that range of WAL records, but
excluding blocks which were removed by truncation after they were
modified and never subsequently modified again. In other words, it
tells us which blocks need to copied in case of an incremental backup
covering that range of WAL records.

A new parameter summarize_wal enables or disables this new background
process.  The background process also automatically deletes summary
files that are older than wal_summarize_keep_time, if that parameter
has a non-zero value and the summarizer is configured to run.

Patch by me, with some design help from Dilip Kumar.  Reviewed by
Matthias van de Meent, Dilip Kumar, Jakub Wartak, Peter Eisentraut,
and Álvaro Herrera.
30 files changed:
doc/src/sgml/config.sgml
src/backend/access/transam/xlog.c
src/backend/backup/Makefile
src/backend/backup/meson.build
src/backend/backup/walsummary.c [new file with mode: 0644]
src/backend/backup/walsummaryfuncs.c [new file with mode: 0644]
src/backend/postmaster/Makefile
src/backend/postmaster/auxprocess.c
src/backend/postmaster/meson.build
src/backend/postmaster/postmaster.c
src/backend/postmaster/walsummarizer.c [new file with mode: 0644]
src/backend/storage/lmgr/lwlocknames.txt
src/backend/utils/activity/pgstat_io.c
src/backend/utils/activity/wait_event_names.txt
src/backend/utils/init/miscinit.c
src/backend/utils/misc/guc_tables.c
src/backend/utils/misc/postgresql.conf.sample
src/bin/initdb/initdb.c
src/common/Makefile
src/common/blkreftable.c [new file with mode: 0644]
src/common/meson.build
src/include/access/xlog.h
src/include/backup/walsummary.h [new file with mode: 0644]
src/include/catalog/pg_proc.dat
src/include/common/blkreftable.h [new file with mode: 0644]
src/include/miscadmin.h
src/include/postmaster/walsummarizer.h [new file with mode: 0644]
src/include/storage/proc.h
src/include/utils/guc_tables.h
src/tools/pgindent/typedefs.list