Thomas Munro [Mon, 7 Sep 2020 06:11:46 +0000 (18:11 +1200)]
Skip unnecessary stat() calls in walkdir().
Some kernels can tell us the type of a "dirent", so we can avoid a call
to stat() or lstat() in many cases. Define a new function
get_dirent_type() to contain that logic, for use by the backend and
frontend versions of walkdir(), and perhaps other callers in future.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKG%2BFzxupGGN4GpUdbzZN%2Btn6FQPHo8w0Q%2BAPH5Wz8RG%2Bww%40mail.gmail.com
Michael Paquier [Mon, 7 Sep 2020 05:34:59 +0000 (14:34 +0900)]
doc: Tweak sentence for pg_checksums when enabling checksums
The previous version of the docs mentioned that files are rewritten,
implying that a second copy of each file gets created, but each file is
updated in-place.
Author: Michael Banck
Reviewed-by: Daniel Gustafsson, Michael Paquier
Discussion: https://postgr.es/m/
858086b6a42fb7d17995b6175856f7e7ec44d0a2.camel@credativ.de
Backpatch-through: 12
Amit Kapila [Mon, 7 Sep 2020 02:38:58 +0000 (08:08 +0530)]
Add additional tests to test streaming of in-progress transactions.
This covers the functionality tests for streaming in-progress
subtransactions, streaming transactions containing rollback to savepoints,
and streaming transactions having DDLs.
Author: Tomas Vondra, Amit Kapila and Dilip Kumar
Reviewed-by: Dilip Kumar
Discussion: https://postgr.es/m/
688b0b7f-2f6c-d827-c27b-
216a8e3ea700@2ndquadrant.com
Tom Lane [Mon, 7 Sep 2020 01:40:39 +0000 (21:40 -0400)]
Apply auto-vectorization to the inner loop of numeric multiplication.
Compile numeric.c with -ftree-vectorize where available, and adjust
the innermost loop of mul_var() so that it is amenable to being
auto-vectorized. (Mainly, that involves making it process the arrays
left-to-right not right-to-left.)
Applying -ftree-vectorize actually makes numeric.o smaller, at least
with my compiler (gcc 8.3.1 on x86_64), and it's a little faster too.
Independently of that, fixing the inner loop to be vectorizable also
makes things a bit faster. But doing both is a huge win for
multiplications with lots of digits. For me, the numeric regression
test is the same speed to within measurement noise, but numeric_big
is a full 45% faster.
We also looked into applying -funroll-loops, but that makes numeric.o
bloat quite a bit, and the additional speed improvement is very
marginal.
Amit Khandekar, reviewed and edited a little by me
Discussion: https://postgr.es/m/CAJ3gD9evtA_vBo+WMYMyT-u=keHX7-r8p2w7OSRfXf42LTwCZQ@mail.gmail.com
Tom Lane [Mon, 7 Sep 2020 01:28:16 +0000 (21:28 -0400)]
Split Makefile symbol CFLAGS_VECTOR into two symbols.
Replace CFLAGS_VECTOR with CFLAGS_UNROLL_LOOPS and CFLAGS_VECTORIZE,
allowing us to distinguish whether we want to apply -funroll-loops,
-ftree-vectorize, or both to a particular source file. Up to now
the only consumer of the symbol has been checksum.c which wants
both, so that there was no need to distinguish; but that's about
to change.
Amit Khandekar, reviewed and edited a little by me
Discussion: https://postgr.es/m/CAJ3gD9evtA_vBo+WMYMyT-u=keHX7-r8p2w7OSRfXf42LTwCZQ@mail.gmail.com
Tom Lane [Sun, 6 Sep 2020 18:13:02 +0000 (14:13 -0400)]
Remove arbitrary line length limits in pg_regress (plain and ECPG).
Refactor replace_string() to use a StringInfo for the modifiable
string argument. This allows the string to be of indefinite size
initially and/or grow substantially during replacement. The previous
logic in convert_sourcefiles_in() had a hard-wired limit of 1024
bytes on any line in input/*.sql or output/*.out files. While we've
not had reports of trouble yet, it'd surely have bit us someday.
This also fixes replace_string() so it won't get into an infinite
loop if the string-to-be-replaced is a substring of the replacement.
That's unlikely to happen in current usage, but the function surely
shouldn't depend on it.
Also fix ecpg_filter() to use a StringInfo and thereby remove its
hard limit of 300 bytes on the length of an ecpg source line.
Asim Rama Praveen and Georgios Kokolatos,
reviewed by Alvaro Herrera and myself
Discussion: https://postgr.es/m/y9Dlk2QhiZ39DhaB1QE9mgZ95HcOQKZCNtGwN7XCRKMdBRBnX_0woaRUtTjloEp4PKA6ERmcUcfq3lPGfKPOJ5xX2TV-5WoRYyySeNHRzdw=@protonmail.com
Tom Lane [Sun, 6 Sep 2020 17:57:10 +0000 (13:57 -0400)]
Refactor pg_get_line() to expose an alternative StringInfo-based API.
Letting the caller provide a StringInfo to read into is helpful when
the caller needs to merge lines or otherwise modify the data after
it's been read. Notably, now the code added by commit
8f8154a50
can use pg_get_line_append() instead of having its own copy of that
logic. A follow-on commit will also make use of this.
Also, since StringInfo buffers are a minimum of 1KB long, blindly
using pg_get_line() in a loop can eat a lot more memory than one would
expect. I discovered for instance that commit
e0f05cd5b caused initdb
to consume circa 10MB to read postgres.bki, even though that's under
1MB worth of data. A less memory-hungry alternative is to re-use the
same StringInfo for all lines and pg_strdup the results.
Discussion: https://postgr.es/m/
1315832.
1599345736@sss.pgh.pa.us
Magnus Hagander [Sun, 6 Sep 2020 17:28:32 +0000 (19:28 +0200)]
Change path in example of file_fdw for logs
It's better to use a relative path into the data directory, than to a
hardcoded home directory of user 'josh'.
Discussion: https://postgr.es/m/CABUevEyuf67Yu_r9gpDMs5MKifK7+-+pe=ZjKzya4JEn9kUk1w@mail.gmail.com
Magnus Hagander [Sun, 6 Sep 2020 17:26:55 +0000 (19:26 +0200)]
Fix typo in comment
Author: Hou, Zhijie
Tom Lane [Sun, 6 Sep 2020 16:55:13 +0000 (12:55 -0400)]
Fix misleading error message about inconsistent moving-aggregate types.
We reported the wrong types when complaining that an aggregate's
moving-aggregate implementation is inconsistent with its regular
implementation.
This was wrong since the feature was introduced, so back-patch
to all supported branches.
Jeff Janes
Discussion: https://postgr.es/m/CAMkU=1x808LH=LPhZp9mNSP0Xd1xDqEd+XeGcvEe48dfE6xV=A@mail.gmail.com
Tom Lane [Sun, 6 Sep 2020 15:50:40 +0000 (11:50 -0400)]
Remove useless lstat() call in pg_rewind.
This is duplicative of an lstat that was just done by the calling
function (traverse_datadir), besides which we weren't really doing
anything with the results. There's not much point in checking to
see if someone removed the file since the previous lstat, since the
FILE_ACTION_REMOVE code would have to deal with missing-file cases
anyway. Moreover, the "exists = false" assignment was a dead store;
nothing was done with that value later.
A syscall saved is a syscall earned, so back-patch to 9.5
where this code was introduced.
Discussion: https://postgr.es/m/
1221796.
1599329320@sss.pgh.pa.us
Peter Eisentraut [Sun, 6 Sep 2020 14:46:13 +0000 (16:46 +0200)]
doc: Don't hide the "Up" link when it is the same as "Home"
The original stylesheets seemed to think this was a good idea, but our
users find it confusing and unhelpful, so undo that logic.
Reported-by: Fabien COELHO <coelho@cri.ensmp.fr>
Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.22.394.
2006210914370.859381%40pseudo
Peter Eisentraut [Sun, 6 Sep 2020 07:32:16 +0000 (09:32 +0200)]
Remove unused parameter
unused since
84d723b6cefcf25b8c800f8aa6cf3c9538a546b4
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Tom Lane [Sat, 5 Sep 2020 20:20:04 +0000 (16:20 -0400)]
Improve some ancient, crufty code in bootstrap + initdb.
At some point back in the last century, somebody felt that reading
all of pg_type twice was cheaper, or at least easier, than using
repalloc() to resize the Typ[] array dynamically. That seems like an
entirely wacko proposition, so rewrite the code to do it the other
way. (To add insult to injury, there were two not-quite-identical
copies of said code.)
initdb.c's readfile() function had the same disease of preferring
to do double the I/O to avoid resizing its output array. Here,
we can make things easier by using the just-invented pg_get_line()
function to handle reading individual lines without a predetermined
notion of how long they are.
On my machine, it's difficult to detect any net change in the
overall runtime of initdb from these changes; but they should
help on slower buildfarm machines (especially since a buildfarm
cycle involves a lot of initdb's these days).
My attention was drawn to these places by scan-build complaints,
but on inspection they needed a lot more work than just suppressing
dead stores :-(
Tom Lane [Sat, 5 Sep 2020 17:17:32 +0000 (13:17 -0400)]
Yet more elimination of dead stores and useless initializations.
I'm not sure what tool Ranier was using, but the ones I contributed
were found by using a newer version of scan-build than I tried before.
Ranier Vilela and Tom Lane
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Michael Paquier [Sat, 5 Sep 2020 12:33:53 +0000 (21:33 +0900)]
Switch to multi-inserts when registering dependencies for many code paths
This commit improves the dependency registrations by taking advantage of
the preliminary work done in
63110c62, to group together the insertion
of dependencies of the same type to pg_depend. With the current layer
of routines available, and as only dependencies of the same type can be
grouped, there are code paths still doing more than one multi-insert
when it is necessary to register dependencies of multiple types
(constraint and index creation are two cases doing that).
While on it, this refactors some of the code to use ObjectAddressSet()
when manipulating object addresses.
Author: Daniel Gustafsson, Michael Paquier
Reviewed-by: Andres Freund, Álvaro Herrera
Discussion: https://postgr.es/m/
20200807061619.GA23955@paquier.xyz
Peter Eisentraut [Sat, 5 Sep 2020 11:28:05 +0000 (13:28 +0200)]
Extend SQL function tests lightly
The basic tests that defined SQL functions didn't actually run the
functions to see if they worked. Add that, and also fix a minor
mistake in a function that was revealed by this. (This is not a
question of test coverage, since there are other places where SQL
functions are run, but it is a bit of a silly test design.)
Discussion: https://www.postgresql.org/message-id/flat/
1c11f1eb-f00c-43b7-799d-
2d44132c02d7@2ndquadrant.com
Peter Eisentraut [Sat, 5 Sep 2020 09:32:20 +0000 (11:32 +0200)]
Fix typo in comment
Michael Paquier [Sat, 5 Sep 2020 04:52:47 +0000 (13:52 +0900)]
Use multi-inserts for pg_depend
This is a follow-up of the work done in
e3931d01. This case is a bit
different than pg_attribute and pg_shdepend: the maximum number of items
to insert is known in advance, but there is no need to handle pinned
dependencies. Hence, the base allocation for slots is done based on the
number of items and the maximum allowed with a cap at 64kB. Slots are
initialized once used to minimize the overhead of the operation.
The insertions can be done for dependencies of the same type. More
could be done by grouping the insertion of multiple dependency types in
a single batch. This is left as future work.
Some of the multi-insert logic is also simplified for pg_shdepend, as
per the feedback discussed for this specific patch. This also moves to
indexing.h the variable capping the maximum amount of data that can be
used at once for a multi-insert, instead of having separate definitions
for pg_attribute, pg_depend and pg_shdepend.
Author: Daniel Gustafsson, Michael Paquier
Reviewed-by: Andres Freund, Álvaro Herrera
Discussion: https://postgr.es/m/
20200807061619.GA23955@paquier.xyz
Tom Lane [Sat, 5 Sep 2020 01:01:58 +0000 (21:01 -0400)]
Make new authentication test case more robust.
I happened to notice that the new test case I added in
b55b4dad9
falls over if one runs "make check" repeatedly; though not in branches
after v10. That's because it was assuming that tmp_check/pgpass
wouldn't exist already. However, it's only been since v11 that the
Makefiles forcibly remove all of tmp_check/ before starting a TAP run.
This fix to unlink the file is therefore strictly necessary only in
v10 ... but it seems wisest to do it across the board, rather than
let the test rely on external logic to get the conditions right.
Tom Lane [Sat, 5 Sep 2020 00:20:05 +0000 (20:20 -0400)]
Fix over-eager ping'ing in logical replication receiver.
Commit
3f60f690f only partially fixed the broken-status-tracking
issue in LogicalRepApplyLoop: we need ping_sent to have the same
lifetime as last_recv_timestamp. The effects are much less serious
than what that commit fixed, though. AFAICS this would just lead to
extra ping requests being sent, once per second until the sender
responds. Still, it's a bug, so backpatch to v10 as before.
Discussion: https://postgr.es/m/959627.
1599248476@sss.pgh.pa.us
Tom Lane [Fri, 4 Sep 2020 22:17:47 +0000 (18:17 -0400)]
Remove still more useless assignments.
Fix some more things scan-build pointed to as dead stores. In some of
these cases, rearranging the code a little leads to more readable
code IMO. It's all cosmetic, though.
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Jeff Davis [Fri, 4 Sep 2020 19:01:58 +0000 (12:01 -0700)]
Fix bogus MaxAllocSize check in logtape.c.
Reported-by: Peter Geoghegan
Discussion: https://postgr.es/m/CAH2-Wz=NZPZc3-fkdmvu=w2itx0PiB-G6QpxHXZOjuvFAzPdZw@mail.gmail.com
Backpatch-through: 13
Alvaro Herrera [Fri, 4 Sep 2020 18:58:32 +0000 (14:58 -0400)]
Report expected contrecord length on mismatch
When reading a WAL record fails to find continuation record(s) of the
proper length, report what it expects, for clarity.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/
20200903212152.GA15319@alvherre.pgsql
Tom Lane [Fri, 4 Sep 2020 18:32:10 +0000 (14:32 -0400)]
Remove some more useless assignments.
Found with clang's scan-build tool. It also whines about a lot of
other dead stores that we should *not* change IMO, either as a matter
of style or future-proofing. But these places seem like clear
oversights.
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Andrew Dunstan [Fri, 4 Sep 2020 17:53:09 +0000 (13:53 -0400)]
Collect attribute data on extension owned tables being dumped
If this data is not collected, pg_dump segfaults if asked for column
inserts.
Fix by Fabrízio de Royes Mello
Backpatch to release 12 where the bug was introduced.
Bruce Momjian [Fri, 4 Sep 2020 17:27:52 +0000 (13:27 -0400)]
C comment: correct use of 64-"byte" cache line size
Reported-by: Kelly Min
Discussion: https://postgr.es/m/CAPSbxatOiQO90LYpSC3+svAU9-sHgDfEP4oFhcEUt_X=DqFA9g@mail.gmail.com
Backpatch-through: 9.5
Tom Lane [Fri, 4 Sep 2020 16:40:28 +0000 (12:40 -0400)]
Fix rare deadlock failure in create_am regression test.
The "DROP ACCESS METHOD gist2" test will require locking the index
to be dropped and then its table; while most ordinary operations
lock a table first then its index. While no concurrent test scripts
should be touching fast_emp4000, autovacuum might chance to be
processing that table when the DROP runs, resulting in a deadlock
failure. This is pretty rare but we see it in the buildfarm from
time to time.
To fix, acquire a lock on fast_emp4000 before issuing the DROP.
Since the point of the exercise is mostly to prevent buildfarm
failures, back-patch to 9.6 where this test was introduced.
Discussion: https://postgr.es/m/839004.
1599185607@sss.pgh.pa.us
Peter Eisentraut [Fri, 4 Sep 2020 06:45:57 +0000 (08:45 +0200)]
doc: Change table alias names to lower case in tutorial chapter
This is needlessly different from our usual style otherwise.
Author: Jürgen Purtz <juergen@purtz.de>
Discussion: https://www.postgresql.org/message-id/flat/
158996922318.7035.
10603922579567326239@wrigleys.postgresql.org
Peter Eisentraut [Fri, 4 Sep 2020 06:39:01 +0000 (08:39 +0200)]
doc: Fix whitespace issue in PDF
Move <indexterm> outside of <para> to avoid whitespace issue in PDF
output.
Author: Jürgen Purtz <juergen@purtz.de>
Discussion: https://www.postgresql.org/message-id/flat/
158996922318.7035.
10603922579567326239@wrigleys.postgresql.org
Peter Eisentraut [Fri, 4 Sep 2020 06:19:20 +0000 (08:19 +0200)]
doc: Use tags consistently in the tutorial chapter
Make more consistent use of <screen> and <programlisting>.
Author: Jürgen Purtz <juergen@purtz.de>
Discussion: https://www.postgresql.org/message-id/flat/
158996922318.7035.
10603922579567326239@wrigleys.postgresql.org
Peter Eisentraut [Fri, 4 Sep 2020 06:02:58 +0000 (08:02 +0200)]
Remove unused parameter
unused since
93ee38eade1b2b4964354b95b01b09e17d6f098d
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Amit Kapila [Fri, 4 Sep 2020 05:55:16 +0000 (11:25 +0530)]
Fix inline marking introduced in commit
464824323e.
Forgot to add inline marking in changes_filename() declaration. In the passing, add
inline marking for a similar function subxact_filename().
Reported-By: Nathan Bossart
Discussion: https://postgr.es/m/
E98FBE8F-B878-480D-A728-
A60C6EED3047@amazon.com
Bruce Momjian [Fri, 4 Sep 2020 02:57:35 +0000 (22:57 -0400)]
remove redundant initializations
Reported-by: Ranier Vilela
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Author: Ranier Vilela
Backpatch-through: master
Michael Paquier [Fri, 4 Sep 2020 01:36:35 +0000 (10:36 +0900)]
Remove variable "concurrent" from ReindexStmt
This node already handles multiple options using a bitmask, so having a
separate boolean flag is not necessary. This simplifies the code a bit
with less arguments to give to the reindex routines, by replacing the
boolean with an equivalent bitmask value.
Reviewed-by: Julien Rouhaud
Discussion: https://postgr.es/m/
20200902110326.GA14963@paquier.xyz
Tom Lane [Fri, 4 Sep 2020 00:09:18 +0000 (20:09 -0400)]
Remove arbitrary restrictions on password length.
This patch started out with the goal of harmonizing various arbitrary
limits on password length, but after awhile a better idea emerged:
let's just get rid of those fixed limits.
recv_password_packet() has an arbitrary limit on the packet size,
which we don't really need, so just drop it. (Note that this doesn't
really affect anything for MD5 or SCRAM password verification, since
those will hash the user's password to something shorter anyway.
It does matter for auth methods that require a cleartext password.)
Likewise remove the arbitrary error condition in pg_saslprep().
The remaining limits are mostly in client-side code that prompts
for passwords. To improve those, refactor simple_prompt() so that
it allocates its own result buffer that can be made as big as
necessary. Actually, it proves best to make a separate routine
pg_get_line() that has essentially the semantics of fgets(), except
that it allocates a suitable result buffer and hence will never
return a truncated line. (pg_get_line has a lot of potential
applications to replace randomly-sized fgets buffers elsewhere,
but I'll leave that for another patch.)
I built pg_get_line() atop stringinfo.c, which requires moving
that code to src/common/; but that seems fine since it was a poor
fit for src/port/ anyway.
This patch is mostly mine, but it owes a good deal to Nathan Bossart
who pressed for a solution to the password length problem and
created a predecessor patch. Also thanks to Peter Eisentraut and
Stephen Frost for ideas and discussion.
Discussion: https://postgr.es/m/
09512C4F-8CB9-4021-B455-
EF4C4F0D55A0@amazon.com
Tom Lane [Thu, 3 Sep 2020 20:52:09 +0000 (16:52 -0400)]
Avoid lockup of a parallel worker when reporting a long error message.
Because sigsetjmp() will restore the initial state with signals blocked,
the code path in bgworker.c for reporting an error and exiting would
execute that way. Usually this is fairly harmless; but if a parallel
worker had an error message exceeding the shared-memory communication
buffer size (16K) it would lock up, because it would wait for a
resume-sending signal from its parallel leader which it would never
detect.
To fix, just unblock signals at the appropriate point.
This can be shown to fail back to 9.6. The lack of parallel query
infrastructure makes it difficult to provide a simple test case for
9.5; but I'm pretty sure the issue exists in some form there as well,
so apply the code change there too.
Vignesh C, reviewed by Bharath Rupireddy, Robert Haas, and myself
Discussion: https://postgr.es/m/CALDaNm1d1hHPZUg3xU4XjtWBOLCrA+-2cJcLpw-cePZ=GgDVfA@mail.gmail.com
Tom Lane [Thu, 3 Sep 2020 16:16:48 +0000 (12:16 -0400)]
Allow records to span multiple lines in pg_hba.conf and pg_ident.conf.
A backslash at the end of a line now causes the next line to be appended
to the current one (effectively, the backslash and newline are discarded).
This allows long HBA entries to be created without legibility problems.
While we're here, get rid of the former hard-wired length limit on
pg_hba.conf lines, by using an expansible StringInfo buffer instead
of a fixed-size local variable.
Since the same code is used to read the ident map file, these changes
apply there as well.
Fabien Coelho, reviewed by Justin Pryzby and David Zhang
Discussion: https://postgr.es/m/alpine.DEB.2.21.
2003251906140.15243@pseudo
Tom Lane [Thu, 3 Sep 2020 15:45:26 +0000 (11:45 -0400)]
Doc: mention packager-supplied tools for server start/stop, initdb, etc.
The majority of our audience is probably using a pre-packaged Postgres
build rather than raw sources. For them, much of runtime.sgml is not
too relevant, and they should be reading the packager's docs instead.
Add some notes pointing that way in appropriate places.
Text by me; thanks to Daniel Gustafsson for review and discussion,
and to Laurenz Albe for an earlier version.
Discussion: https://postgr.es/m/
159430831443.16535.
11360317280100947016@wrigleys.postgresql.org
Peter Eisentraut [Thu, 3 Sep 2020 11:15:53 +0000 (13:15 +0200)]
doc: Make SQL command names in the catalog documentation links
In passing, fix the initdb references to be <application> rather than
<command>, which is what we normally use.
Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Discussion: https://www.postgresql.org/message-id/flat/87mu5xqc11.fsf@wibble.ilmari.org
Peter Eisentraut [Thu, 3 Sep 2020 11:15:53 +0000 (13:15 +0200)]
doc: Add missing cross-links in system catalog documentation
This makes the first mention of a system catalog or view in each
paragraph in the system system catalog and view documentation pages
hyperlinks, for easier navigation.
Also linkify the first mention of pg_hba.conf in pg_hba_file_rules, as
that's more specific and easier to spot than the link to the client
authentication chapter.
Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Discussion: https://www.postgresql.org/message-id/flat/87mu5xqc11.fsf@wibble.ilmari.org
Peter Eisentraut [Thu, 3 Sep 2020 10:38:32 +0000 (12:38 +0200)]
Fix XML id to match containing page
This was apparently a typo when this part of the documentation was
first added.
Amit Kapila [Thu, 3 Sep 2020 02:24:07 +0000 (07:54 +0530)]
Add support for streaming to built-in logical replication.
To add support for streaming of in-progress transactions into the
built-in logical replication, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
Author: Tomas Vondra, Dilip Kumar and Amit Kapila
Reviewed-by: Amit Kapila, Kuntal Ghosh and Ajin Cherian
Tested-by: Neha Sharma, Mahendra Singh Thalor and Ajin Cherian
Discussion: https://postgr.es/m/
688b0b7f-2f6c-d827-c27b-
216a8e3ea700@2ndquadrant.com
Tom Lane [Wed, 2 Sep 2020 22:23:56 +0000 (18:23 -0400)]
Add string_to_table() function.
This splits a string at occurrences of a delimiter. It is exactly like
string_to_array() except for producing a set of values instead of an
array of values. Thus, the relationship of these two functions is
the same as between regexp_split_to_table() and regexp_split_to_array().
Although the same results could be had from unnest(string_to_array()),
this is somewhat faster than that, and anyway it seems reasonable to
have it for symmetry with the regexp functions.
Pavel Stehule, reviewed by Peter Smith
Discussion: https://postgr.es/m/CAFj8pRD8HOpjq2TqeTBhSo_QkzjLOhXzGCpKJ4nCs7Y9SQkuPw@mail.gmail.com
Peter Eisentraut [Wed, 2 Sep 2020 13:17:33 +0000 (15:17 +0200)]
Remove unused parameter
unused since
39bd3fd1db6f3aa3764d4a1bebcd71c4e9c00281
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Michael Paquier [Wed, 2 Sep 2020 07:59:22 +0000 (16:59 +0900)]
Add access method names to \d[i|m|t]+ in psql
Listing a full set of relations with those psql meta-commands, without a
matching pattern, has never showed the access method associated with
each relation. This commit adds the access method of tables, indexes
and matviews, masking it for relation kinds where it does not apply.
Note that when HIDE_TABLEAM is enabled, the information does not show
up. This is available when connecting to a backend version of at least
12, where table AMs have been introduced.
Author: Georgios Kokolatos
Reviewed-by: Vignesh C, Michael Paquier, Justin Pryzby
Discussion: https://postgr.es/m/svaS1VTOEscES9CLKVTeKItjJP1EEJuBhTsA0ESOdlnbXeQSgycYwVlliL5zt8Jwcfo4ATYDXtEqsExxjkSkkhCSTCL8fnRgaCAJdr0unUg=@protonmail.com
Michael Paquier [Wed, 2 Sep 2020 05:56:59 +0000 (14:56 +0900)]
Fix thinko with definition of REINDEXOPT_MISSING_OK
This had no direct consequences, but let's be consistent and it would be
confusing when adding new flags. Oversight in
1d65416.
Reported-by: Justin Pryzby
Discussion: https://postgr.es/m/
20200902024148.GB20149@telsasoft.com
Fujii Masao [Wed, 2 Sep 2020 01:55:55 +0000 (10:55 +0900)]
Avoid unnecessary acquisition of SyncRepLock in transaction commit time.
In SyncRepWaitForLSN() routine called in transaction commit time,
SyncRepLock is necessary to atomically both check the shared
sync_standbys_defined flag and operate the sync replication wait-queue.
On the other hand, when the flag is false, the lock is not necessary
because the wait-queue is not touched. But due to the changes by
commit
48c9f49265, previously the lock was taken whatever the flag was.
This could cause unnecessary performance overhead in every transaction
commit time. Therefore this commit avoids that unnecessary aquisition
of SyncRepLock.
Author: Fujii Masao
Reviewed-by: Asim Praveen, Masahiko Sawada,
Discussion: https://postgr.es/m/
20200406050332.nsscfqjzk2d57zyx@alap3.anarazel.de
Alvaro Herrera [Wed, 2 Sep 2020 00:43:23 +0000 (20:43 -0400)]
Fix typo in comment
Introduced by
8b08f7d4820f; backpatch to 11.
Discussion: https://postgr.es/m/
20200812214918.GA30353@alvherre.pgsql
Michael Paquier [Wed, 2 Sep 2020 00:08:12 +0000 (09:08 +0900)]
Improve handling of dropped relations for REINDEX DATABASE/SCHEMA/SYSTEM
When multiple relations are reindexed, a scan of pg_class is done first
to build the list of relations to work on. However the REINDEX logic
has never checked if a relation listed still exists when beginning the
work on it, causing for example sudden cache lookup failures.
This commit adds safeguards against dropped relations for REINDEX,
similarly to VACUUM or CLUSTER where we try to open the relation,
ignoring it if it is missing. A new option is added to the REINDEX
routines to control if a missed relation is OK to ignore or not.
An isolation test, based on REINDEX SCHEMA, is added for the concurrent
and non-concurrent cases.
Author: Michael Paquier
Reviewed-by: Anastasia Lubennikova
Discussion: https://postgr.es/m/
20200813043805.GE11663@paquier.xyz
Tom Lane [Tue, 1 Sep 2020 22:40:37 +0000 (18:40 -0400)]
Improve test coverage of ginvacuum.c.
Add a test case that exercises vacuum's deletion of empty GIN
posting pages. Since this is a temp table, it should now work
reliably to delete a bunch of rows and immediately VACUUM.
Before the preceding commit, this would not have had the desired
effect, at least not in parallel regression tests.
Discussion: https://postgr.es/m/
3490536.
1598629609@sss.pgh.pa.us
Tom Lane [Tue, 1 Sep 2020 22:37:12 +0000 (18:37 -0400)]
Set cutoff xmin more aggressively when vacuuming a temporary table.
Since other sessions aren't allowed to look into a temporary table
of our own session, we do not need to worry about the global xmin
horizon when setting the vacuum XID cutoff. Indeed, if we're not
inside a transaction block, we may set oldestXmin to be the next
XID, because there cannot be any in-doubt tuples in a temp table,
nor any tuples that are dead but still visible to some snapshot of
our transaction. (VACUUM, of course, is never inside a transaction
block; but we need to test that because CLUSTER shares the same code.)
This approach allows us to always clean out a temp table completely
during VACUUM, independently of concurrent activity. Aside from
being useful in its own right, that simplifies building reproducible
test cases.
Discussion: https://postgr.es/m/
3490536.
1598629609@sss.pgh.pa.us
Bruce Momjian [Tue, 1 Sep 2020 21:00:10 +0000 (17:00 -0400)]
doc: clarify that max_wal_size is "during" checkpoints
Previous wording was "between".
Reported-by: Pavel Luzanov
Discussion: https://postgr.es/m/
26906a54-d7cb-2f8e-eed7-
e31660024694@postgrespro.ru
Backpatch-through: 9.5
Alvaro Herrera [Tue, 1 Sep 2020 17:40:43 +0000 (13:40 -0400)]
Raise error on concurrent drop of partitioned index
We were already raising an error for DROP INDEX CONCURRENTLY on a
partitioned table, albeit a different and confusing one:
ERROR: DROP INDEX CONCURRENTLY must be first action in transaction
Change that to throw a more comprehensible error:
ERROR: cannot drop partitioned index \"%s\" concurrently
Michael Paquier authored the test case for indexes on temporary
partitioned tables.
Backpatch to 11, where indexes on partitioned tables were added.
Reported-by: Jan Mussler <jan.mussler@zalando.de>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/16594-
d2956ca909585067@postgresql.org
Tom Lane [Tue, 1 Sep 2020 17:14:44 +0000 (13:14 -0400)]
Teach libpq to handle arbitrary-length lines in .pgpass files.
Historically there's been a hard-wired assumption here that no line of
a .pgpass file could be as long as NAMEDATALEN*5 bytes. That's a bit
shaky to start off with, because (a) there's no reason to suppose that
host names fit in NAMEDATALEN, and (b) this figure fails to allow for
backslash escape characters. However, it fails completely if someone
wants to use a very long password, and we're now hearing reports of
people wanting to use "security tokens" that can run up to several
hundred bytes. Another angle is that the file is specified to allow
comment lines, but there's no reason to assume that long comment lines
aren't possible.
Rather than guessing at what might be a more suitable limit, let's
replace the fixed-size buffer with an expansible PQExpBuffer. That
adds one malloc/free cycle to the typical use-case, but that's surely
pretty cheap relative to the I/O this code has to do.
Also, add TAP test cases to exercise this code, because there was no
test coverage before.
This reverts most of commit
2eb3bc588, as there's no longer a need for
a warning message about overlength .pgpass lines. (I kept the explicit
check for comment lines, though.)
In HEAD and v13, this also fixes an oversight in
74a308cf5: there's not
much point in explicit_bzero'ing the line buffer if we only do so in two
of the three exit paths.
Back-patch to all supported branches, except that the test case only
goes back to v10 where src/test/authentication/ was added.
Discussion: https://postgr.es/m/
4187382.
1598909041@sss.pgh.pa.us
Amit Kapila [Tue, 1 Sep 2020 02:41:39 +0000 (08:11 +0530)]
Fix the SharedFileSetUnregister API.
Commit
808e13b282 introduced a few APIs to extend the existing Buffile
interface. In SharedFileSetDeleteOnProcExit, it tries to delete the list
element while traversing the list with 'foreach' construct which makes the
behavior of list traversal unpredictable.
Author: Amit Kapila
Reviewed-by: Dilip Kumar
Tested-by: Dilip Kumar and Neha Sharma
Discussion: https://postgr.es/m/CAA4eK1JhLatVcQ2OvwA_3s0ih6Hx9+kZbq107cXVsSWWukH7vA@mail.gmail.com
Bruce Momjian [Mon, 31 Aug 2020 22:48:38 +0000 (18:48 -0400)]
doc: document how the backup manifest is transferred
Reported-by: Bernd Helmle
Discussion: https://postgr.es/m/
31acf8b0f1f701d53245e0cae38abdf5c3a0d559.camel@oopsware.de
Backpatch-through: 13
Bruce Momjian [Mon, 31 Aug 2020 22:33:37 +0000 (18:33 -0400)]
doc: add commas after 'i.e.' and 'e.g.'
This follows the American format,
https://jakubmarian.com/comma-after-i-e-and-e-g/. There is no intention
of requiring this format for future text, but making existing text
consistent every few years makes sense.
Discussion: https://postgr.es/m/
20200825183619.GA22369@momjian.us
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 21:36:23 +0000 (17:36 -0400)]
pg_upgrade doc: mention saving postgresql.conf.auto files
Also mention files included by postgresql.conf.
Reported-by: Álvaro Herrera
Discussion: https://postgr.es/m/
08AD4526-75AB-457B-B2DD-
099663F28040@yesql.se
Backpatch-through: 9.5
Alvaro Herrera [Mon, 31 Aug 2020 21:04:40 +0000 (17:04 -0400)]
doc: Update partitioning limitation on BEFORE triggers
Reported-by: Erwin Brandstetter <brsaweda@gmail.com>
Discussion: https://postgr.es/m/CAGHENJ6Le7S3qJJx2TvWvTwRNS3N=BtoNeb7AF2rZvfNBMeQcg@mail.gmail.com
Bruce Momjian [Mon, 31 Aug 2020 21:05:53 +0000 (17:05 -0400)]
docs: in mapping SQL to C data types, timestamp isn't a pointer
It is an int64.
Reported-by: ajulien@shaktiware.fr
Discussion: https://postgr.es/m/
159845038271.24995.
15682121015698255155@wrigleys.postgresql.org
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 20:59:59 +0000 (16:59 -0400)]
doc: cross-link file-fdw and CSV config log sections
There is an file-fdw example that reads the server config file, so cross
link them.
Reported-by: Oleg Samoilov
Discussion: https://postgr.es/m/
159800192078.2886.
10431506404995508950@wrigleys.postgresql.org
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 20:21:03 +0000 (16:21 -0400)]
docs: clarify intermediate certificate creation instructions
Specifically, explain the v3_ca openssl specification.
Discussion: https://postgr.es/m/
20200824175653.GA32411@momjian.us
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 19:23:19 +0000 (15:23 -0400)]
docs: replace "stable storage" with "durable" in descriptions
For PG, "durable storage" has a clear meaning, while "stable storage"
does not, so use the former.
Discussion: https://postgr.es/m/
20200817165222.GA31806@momjian.us
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 17:58:00 +0000 (13:58 -0400)]
C comment: remove mention of use of t_hoff WAL structure member
Reported-by: Antonin Houska
Discussion: https://postgr.es/m/21643.
1595353537@antos
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 17:49:17 +0000 (13:49 -0400)]
doc: improve description of subscripting of arrays
It wasn't clear the non-integers are cast to integers for subscripting,
rather than throwing an error.
Reported-by: sean@materialize.io
Discussion: https://postgr.es/m/
159538675800.624.
7728794628229799531@wrigleys.postgresql.org
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 17:43:05 +0000 (13:43 -0400)]
docs: improve 'capitals' inheritance example
Adds constraints and improves wording.
Reported-by: 2552891@gmail.com
Discussion: https://postgr.es/m/
159586122762.680.
1361378513036616007@wrigleys.postgresql.org
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 17:20:04 +0000 (13:20 -0400)]
doc: clarify the useful features of procedures
This was not clearly documented when procedures were added in PG 11.
Reported-by: Robin Abbi
Discussion: https://postgr.es/m/CAGmg_NX327KKVuJmbWZD=pGutYFxzZjX1rU+3ji8UuX=8ONn9Q@mail.gmail.com
Backpatch-through: 11
Magnus Hagander [Mon, 31 Aug 2020 11:03:54 +0000 (13:03 +0200)]
Fix docs bug stating file_fdw requires absolute paths
It has always (since the first commit) worked with relative paths, so
use the same wording as other parts of the documentation.
Author: Bruce Momjian
Discussion: https://postgr.es/m/CABUevExx-hm=cit+A9LeKBH39srvk8Y2tEZeEAj5mP8YfzNKUg@mail.gmail.com
Tom Lane [Sun, 30 Aug 2020 18:37:24 +0000 (14:37 -0400)]
Mark factorial operator, and postfix operators in general, as deprecated.
Per discussion, we're planning to remove parser support for postfix
operators in order to simplify the grammar. So it behooves us to
put out a deprecation notice at least one release before that.
There is only one built-in postfix operator, ! for factorial.
Label it deprecated in the docs and in pg_description, and adjust
some examples that formerly relied on it. (The sister prefix
operator !! is also deprecated. We don't really have to remove
that one, but since we're suggesting that people use factorial()
instead, it seems better to remove both operators.)
Also state in the CREATE OPERATOR ref page that postfix operators
in general are going away.
Although this changes the initial contents of pg_description,
I did not force a catversion bump; it doesn't seem essential.
In v13, also back-patch
4c5cf5431, so that there's someplace for
the <link>s to point to.
Mark Dilger and John Naylor, with some adjustments by me
Discussion: https://postgr.es/m/
BE2DF53D-251A-4E26-972F-
930E523580E9@enterprisedb.com
Tom Lane [Sun, 30 Aug 2020 16:21:51 +0000 (12:21 -0400)]
Redefine pg_class.reltuples to be -1 before the first VACUUM or ANALYZE.
Historically, we've considered the state with relpages and reltuples
both zero as indicating that we do not know the table's tuple density.
This is problematic because it's impossible to distinguish "never yet
vacuumed" from "vacuumed and seen to be empty". In particular, a user
cannot use VACUUM or ANALYZE to override the planner's normal heuristic
that an empty table should not be believed to be empty because it is
probably about to get populated. That heuristic is a good safety
measure, so I don't care to abandon it, but there should be a way to
override it if the table is indeed intended to stay empty.
Hence, represent the initial state of ignorance by setting reltuples
to -1 (relpages is still set to zero), and apply the minimum-ten-pages
heuristic only when reltuples is still -1. If the table is empty,
VACUUM or ANALYZE (but not CREATE INDEX) will override that to
reltuples = relpages = 0, and then we'll plan on that basis.
This requires a bunch of fiddly little changes, but we can get rid of
some ugly kluges that were formerly needed to maintain the old definition.
One notable point is that FDWs' GetForeignRelSize methods will see
baserel->tuples = -1 when no ANALYZE has been done on the foreign table.
That seems like a net improvement, since those methods were formerly
also in the dark about what baserel->tuples = 0 really meant. Still,
it is an API change.
I bumped catversion because code predating this change would get confused
by seeing reltuples = -1.
Discussion: https://postgr.es/m/
F02298E0-6EF4-49A1-BCB6-
C484794D9ACC@thebuild.com
Michael Paquier [Sun, 30 Aug 2020 05:14:34 +0000 (14:14 +0900)]
Reset indisreplident for an invalid index in DROP INDEX CONCURRENTLY
A failure when dropping concurrently an index used in a replica identity
could leave in pg_index an index marked as !indisvalid and
indisreplident. Reindexing this index would switch back indisvalid to
true, and if the replica identity of the parent relation was switched to
use a different index, it would be possible to finish with more than one
index marked as indisreplident. If that were to happen, this could mess
up with the relation cache as an incorrect index could be used for the
replica identity.
Indexes marked as invalid are discarded as candidates for the replica
identity, as of RelationGetIndexList(), so similarly to what is done
with indisclustered, resetting indisreplident when the index is marked
as invalid keeps things consistent. REINDEX CONCURRENTLY's swapping
already resets the flag for the old index, while the new index inherits
the value of the old index to-be-dropped, so only DROP INDEX was an
issue.
Even if this is a bug, the sequence able to reproduce a problem requires
a failure while running DROP INDEX CONCURRENTLY, something unlikely
going to happen in the field, so no backpatch is done.
Author: Michael Paquier
Reviewed-by: Dmitry Dolgov
Discussion: https://postgr.es/m/
20200827025721.GN2017@paquier.xyz
Michael Paquier [Fri, 28 Aug 2020 07:54:59 +0000 (16:54 +0900)]
doc: Rework tables for built-in operator classes of index AMs
The tables listing all the operator classes available for BRIN, GIN,
GiST and SP-GiST had a confusing format where the same operator could be
listed multiple times, for different data types. This improves the
shape of these tables by adding the types associated to each operator,
for their associated operator class.
Each table included previously the data type that could be used for an
operator class in an extra column. This is removed to reduce the width
of the tables as this is now described within each operator. This also
makes the tables fit better in the PDF documentation.
Reported-by: osdba
Author: Michael Paquier
Reviewed-by: Álvaro Herrera, Tom Lane, Bruce Momjian
Discussion: https://postgr.es/m/
38d55061.9604.
173b32c60ec.Coremail.mailtch@163.com
Peter Eisentraut [Fri, 28 Aug 2020 06:19:12 +0000 (08:19 +0200)]
doc: Update cracklib URL
Author: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Laurenz Albe <laurenz.albe@cybertec.at>
Discussion: https://www.postgresql.org/message-id/flat/
f7266133-618a-0adc-52ef-
f43c78806b0e%402ndquadrant.com
Peter Eisentraut [Fri, 28 Aug 2020 06:16:32 +0000 (08:16 +0200)]
passwordcheck: Log cracklib diagnostics
When calling cracklib to check the password, the diagnostic from
cracklib was thrown away. This would hide essential information such
as no dictionary being installed. Change this to show the cracklib
error message using errdetail_log().
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Laurenz Albe <laurenz.albe@cybertec.at>
Discussion: https://www.postgresql.org/message-id/flat/
f7266133-618a-0adc-52ef-
f43c78806b0e%402ndquadrant.com
Tom Lane [Thu, 27 Aug 2020 21:36:13 +0000 (17:36 -0400)]
Fix code for re-finding scan position in a multicolumn GIN index.
collectMatchBitmap() needs to re-find the index tuple it was previously
looking at, after transiently dropping lock on the index page it's on.
The tuple should still exist and be at its prior position or somewhere
to the right of that, since ginvacuum never removes tuples but
concurrent insertions could add one. However, there was a thinko in
that logic, to the effect of expecting any inserted tuples to have the
same index "attnum" as what we'd been scanning. Since there's no
physical separation of tuples with different attnums, it's not terribly
hard to devise scenarios where this fails, leading to transient "lost
saved point in index" errors. (While I've duplicated this with manual
testing, it seems impossible to make a reproducible test case with our
available testing technology.)
Fix by just continuing the scan when the attnum doesn't match.
While here, improve the error message used if we do fail, so that it
matches the wording used in btree for a similar case.
collectMatchBitmap()'s posting-tree code path was previously not
exercised at all by our regression tests. While I can't make
a regression test that exhibits the bug, I can at least improve
the code coverage here, so do that. The test case I made for this
is an extension of one added by
4b754d6c1, so it only works in
HEAD and v13; didn't seem worth trying hard to back-patch it.
Per bug #16595 from Jesse Kinkead. This has been broken since
multicolumn capability was added to GIN (commit
27cb66fdf),
so back-patch to all supported branches.
Discussion: https://postgr.es/m/16595-
633118be8eef9ce2@postgresql.org
Michael Paquier [Thu, 27 Aug 2020 07:40:34 +0000 (16:40 +0900)]
Fix comment in procarray.c
The description of GlobalVisDataRels was missing, GlobalVisCatalogRels
being mentioned instead.
Author: Jim Nasby
Discussion: https://postgr.es/m/
8e06c883-2858-1fd4-07c5-
560c28b08dcd@amazon.com
Tom Lane [Wed, 26 Aug 2020 21:08:11 +0000 (17:08 -0400)]
Suppress compiler warning in non-cassert builds.
Oversight in
808e13b28, reported by Bruce Momjian.
Discussion: https://postgr.es/m/
20200826160251.GB21909@momjian.us
Michael Paquier [Wed, 26 Aug 2020 11:42:27 +0000 (20:42 +0900)]
Add regression tests for REPLICA IDENTITY with dropped indexes
REPLICA IDENTITY USING INDEX behaves the same way as NOTHING if the
associated index is dropped, even if there is a primary key that could
be used as a fallback for the changes generated. There have never been
any tests to cover such scenarios, so this commit closes the gap.
Author: Michael Paquier
Reviewed-by: Masahiko Sawada, Rahila Syed, Euler Taveira
Discussion: https://postgr.es/m/
20200522035028.GO2355@paquier.xyz
Amit Kapila [Wed, 26 Aug 2020 04:10:52 +0000 (09:40 +0530)]
Add additional information in the vacuum error context.
The additional information added will be an offset number for heap
operations. This information will help us in finding the exact tuple due
to which the error has occurred.
Author: Mahendra Singh Thalor and Amit Kapila
Reviewed-by: Sawada Masahiko, Justin Pryzby and Amit Kapila
Discussion: https://postgr.es/m/CAKYtNApK488TDF4bMbw+1QH8HJf9cxdNDXquhU50TK5iv_FtCQ@mail.gmail.com
Amit Kapila [Wed, 26 Aug 2020 02:06:43 +0000 (07:36 +0530)]
Extend the BufFile interface.
Allow BufFile to support temporary files that can be used by the single
backend when the corresponding files need to be survived across the
transaction and need to be opened and closed multiple times. Such files
need to be created as a member of a SharedFileSet.
Additionally, this commit implements the interface for BufFileTruncate to
allow files to be truncated up to a particular offset and extends the
BufFileSeek API to support the SEEK_END case. This also adds an option to
provide a mode while opening the shared BufFiles instead of always opening
in read-only mode.
These enhancements in BufFile interface are required for the upcoming
patch to allow the replication apply worker, to handle streamed
in-progress transactions.
Author: Dilip Kumar, Amit Kapila
Reviewed-by: Amit Kapila
Tested-by: Neha Sharma
Discussion: https://postgr.es/m/
688b0b7f-2f6c-d827-c27b-
216a8e3ea700@2ndquadrant.com
Fujii Masao [Wed, 26 Aug 2020 01:52:02 +0000 (10:52 +0900)]
Add regression test for pg_backend_memory_contexts.
Author: Atsushi Torikoshi
Reviewed-by: Michael Paquier, Fujii Masao
Discussion: https://postgr.es/m/
20200819135545.GC19121@paquier.xyz
Fujii Masao [Wed, 26 Aug 2020 01:51:31 +0000 (10:51 +0900)]
Move codes for pg_backend_memory_contexts from mmgr/mcxt.c to adt/mcxtfuncs.c.
Previously the codes for pg_backend_memory_contexts were in
src/backend/utils/mmgr/mcxt.c. This commit moves them to
src/backend/utils/adt/mcxtfuncs.c so that mcxt.c basically includes
only the low-level interface for memory contexts.
Author: Atsushi Torikoshi
Reviewed-by: Michael Paquier, Fujii Masao
Discussion: https://postgr.es/m/
20200819135545.GC19121@paquier.xyz
Fujii Masao [Wed, 26 Aug 2020 01:50:02 +0000 (10:50 +0900)]
Prevent non-superusers from reading pg_backend_memory_contexts, by default.
pg_backend_memory_contexts view contains some internal information of
memory contexts. Since exposing them to any users by default may cause
security issue, this commit allows only superusers to read this view,
by default, like we do for pg_shmem_allocations view.
Bump catalog version.
Author: Atsushi Torikoshi
Reviewed-by: Michael Paquier, Fujii Masao
Discussion: https://postgr.es/m/
1414992.
1597849297@sss.pgh.pa.us
David Rowley [Tue, 25 Aug 2020 22:51:36 +0000 (10:51 +1200)]
Fixup some misusages of bms_num_members()
It's a bit inefficient to test if a Bitmapset is empty by counting all the
members and seeing if that number is zero. It's much better just to use
bms_is_empty(). Likewise for checking if there are at least two members,
just use bms_membership(), which does not need to do anything more after
finding two members.
Discussion: https://postgr.es/m/CAApHDvpvwm_QjbDOb5xga%2BKmX9XkN9xQavNGm3SvDbVnCYOerQ%40mail.gmail.com
Reviewed-by: Tomas Vondra
Bruce Momjian [Tue, 25 Aug 2020 13:53:12 +0000 (09:53 -0400)]
docs: client certificates are always sent to the server
They are not "requested" by the server.
Reported-by: Kyotaro Horiguchi
Discussion: https://postgr.es/m/
20200825.155320.
986648039251743210.horikyota.ntt@gmail.com
Backpatch-through: 9.5
Peter Eisentraut [Tue, 25 Aug 2020 05:29:05 +0000 (07:29 +0200)]
doc: Fix up title case
This fixes some instances that were missed in earlier processings and
that now look a bit strange because they are inconsistent with nearby
titles.
Michael Paquier [Mon, 24 Aug 2020 07:46:52 +0000 (16:46 +0900)]
doc: Fix some markups for support functions of index AMs
All the documentation of index AMs has been using <replaceable> for
local_relopts. This is a structure, so <structname> is a much better
choice.
Alexander has found the inconsistency for btree, while I have spotted
the rest when applying the concept of consistency to the docs.
Author: Alexander Lakhin, Michael Paquier
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/
20200822133022.GC24782@paquier.xyz
Amit Kapila [Mon, 24 Aug 2020 02:46:19 +0000 (08:16 +0530)]
Improve the vacuum error context phase information.
We were displaying the wrong phase information for 'info' message in the
index clean up phase because we were switching to the previous phase a bit
early. We were also not displaying context information for heap phase
unless the block number is valid which is fine for error cases but for
messages at 'info' or lower error level it appears to be inconsistent with
index phase information.
Reported-by: Sawada Masahiko
Author: Sawada Masahiko
Reviewed-by: Amit Kapila
Backpatch-through: 13, where it was introduced
Discussion: https://postgr.es/m/CA+fd4k4HcbhPnCs7paRTw1K-AHin8y4xKomB9Ru0ATw0UeTy2w@mail.gmail.com
Tom Lane [Sat, 22 Aug 2020 18:46:40 +0000 (14:46 -0400)]
Avoid pushing quals down into sub-queries that have grouping sets.
The trouble with doing this is that an apparently-constant subquery
output column isn't really constant if it is a grouping column that
appears in only some of the grouping sets. A qual using such a
column would be subject to incorrect const-folding after push-down,
as seen in bug #16585 from Paul Sivash.
To fix, just disable qual pushdown altogether if the sub-query has
nonempty groupingSets. While we could imagine far less restrictive
solutions, there is not much point in working harder right now,
because subquery_planner() won't move HAVING clauses to WHERE within
such a subquery. If the qual stays in HAVING it's not going to be
a lot more useful than if we'd kept it at the outer level.
Having said that, this restriction could be removed if we used a
parsetree representation that distinguished such outputs from actual
constants, which is something I hope to do in future. Hence, make
the patch a minimal addition rather than integrating it more tightly
(e.g. by renumbering the existing items in subquery_is_pushdown_safe's
comment).
Back-patch to 9.5 where grouping sets were introduced.
Discussion: https://postgr.es/m/16585-
9d8c340d23ade8c1@postgresql.org
Tom Lane [Sat, 22 Aug 2020 16:34:17 +0000 (12:34 -0400)]
Fix ALTER TABLE's scheduling rules for AT_AddConstraint subcommands.
Commit
1281a5c90 rearranged the logic in this area rather drastically,
and it broke the case of adding a foreign key constraint in the same
ALTER that adds the pkey or unique constraint it depends on. While
self-referential fkeys are surely a pretty niche case, this used to
work so we shouldn't break it.
To fix, reorganize the scheduling rules in ATParseTransformCmd so
that a transformed AT_AddConstraint subcommand will be delayed into
a later pass in all cases, not only when it's been spit out as a
side-effect of parsing some other command type.
Also tweak the logic so that we won't run ATParseTransformCmd twice
while doing this. It seems to work even without that, but it's
surely wasting cycles to do so.
Per bug #16589 from Jeremy Evans. Back-patch to v13 where the new
code was introduced.
Discussion: https://postgr.es/m/16589-
31c8d981ca503896@postgresql.org
Michael Paquier [Sat, 22 Aug 2020 13:26:10 +0000 (22:26 +0900)]
doc: Fix format, incorrect structure names and markup inconsistencies
Author: Alexander Lakhin
Discussion: https://postgr.es/m/
a2345841-10a5-4eef-257c-
02302347cf39@gmail.com
Backpatch-through: 13
Bruce Momjian [Sat, 22 Aug 2020 00:23:09 +0000 (20:23 -0400)]
docs: improve description of how to handle multiple databases
This is a redesign of the intro to the managing databases chapter.
Discussion: https://postgr.es/m/
159586122762.680.
1361378513036616007@wrigleys.postgresql.org
Author: David G. Johnston
Backpatch-through: 9.5
Bruce Momjian [Fri, 21 Aug 2020 22:29:37 +0000 (18:29 -0400)]
docs: add COMMENT examples for new features, rename rtree
Reported-by: Jürgen Purtz
Discussion: https://postgr.es/m/
15ec5428-d46a-1725-f38d-
44986a977abb@purtz.de
Author: Jürgen Purtz
Backpatch-through: 11
Tom Lane [Fri, 21 Aug 2020 19:00:42 +0000 (15:00 -0400)]
Fix handling of CREATE TABLE LIKE with inheritance.
If a CREATE TABLE command uses both LIKE and traditional inheritance,
Vars in CHECK constraints and expression indexes that are absorbed
from a LIKE parent table tended to get mis-numbered, resulting in
wrong answers and/or bizarre error messages (though probably not any
actual crashes, thanks to validation occurring in the executor).
In v12 and up, the same could happen to Vars in GENERATED expressions,
even in cases with no LIKE clause but multiple traditional-inheritance
parents.
The cause of the problem for LIKE is that parse_utilcmd.c supposed
it could renumber such Vars correctly during transformCreateStmt(),
which it cannot since we have not yet accounted for columns added via
inheritance. Fix that by postponing processing of LIKE INCLUDING
CONSTRAINTS, DEFAULTS, GENERATED, INDEXES till after we've performed
DefineRelation().
The error with GENERATED and multiple inheritance is a simple oversight
in MergeAttributes(); it knows it has to renumber Vars in inherited
CHECK constraints, but forgot to apply the same processing to inherited
GENERATED expressions (a/k/a defaults).
Per bug #16272 from Tom Gottfried. The non-GENERATED variants of the
issue are ancient, presumably dating right back to the addition of
CREATE TABLE LIKE; hence back-patch to all supported branches.
Discussion: https://postgr.es/m/16272-
6e32da020e9a9381@postgresql.org
Fujii Masao [Fri, 21 Aug 2020 16:22:55 +0000 (01:22 +0900)]
Fix explain regression test failure.
Commit
9d701e624f caused the regression test for EXPLAIN to fail on
the buildfarm member prion. This happened because of instability of
test output, i.e., in text format, whether "Planning:" line is output
varies depending on the system state.
This commit updated the regression test so that it ignores that
"Planning:" line to produce more stable test output and get rid of
the test failure.
Back-patch to v13.
Author: Fujii Masao
Discussion: https://postgr.es/m/
1803897.
1598021621@sss.pgh.pa.us
Fujii Masao [Fri, 21 Aug 2020 11:48:59 +0000 (20:48 +0900)]
Rework EXPLAIN for planner's buffer usage.
Commit
ce77abe63c allowed EXPLAIN (BUFFERS) to report the information
on buffer usage during planning phase. However three issues were
reported regarding this feature.
(1) Previously, EXPLAIN option BUFFERS required ANALYZE. So the query
had to be actually executed by specifying ANALYZE even when we
want to see only the planner's buffer usage. This was inconvenient
especially when the query was write one like DELETE.
(2) EXPLAIN included the planner's buffer usage in summary
information. So SUMMARY option had to be enabled to report that.
Also this format was confusing.
(3) The output structure for planning information was not consistent
between TEXT format and the others. For example, "Planning" tag
was output in JSON format, but not in TEXT format.
For (1), this commit allows us to perform EXPLAIN (BUFFERS) without
ANALYZE to report the planner's buffer usage.
For (2), this commit changed EXPLAIN output so that the planner's
buffer usage is reported before summary information.
For (3), this commit made the output structure for planning
information more consistent between the formats.
Back-patch to v13 where the planner's buffer usage was allowed to
be reported in EXPLAIN.
Reported-by: Pierre Giraud, David Rowley
Author: Fujii Masao
Reviewed-by: David Rowley, Julien Rouhaud, Pierre Giraud
Discussion: https://postgr.es/m/
07b226e6-fa49-687f-b110-
b7c37572f69e@dalibo.com
Fujii Masao [Fri, 21 Aug 2020 03:33:30 +0000 (12:33 +0900)]
Fix typos in comments.
Author: Masahiko Sawada
Reviewed-by: Fujii Masao
Discussion: https://postgr.es/m/CA+fd4k4m9hFSrRLB3etPWO5_v5=MujVZWRtz63q+55hM0Dz25Q@mail.gmail.com
David Rowley [Thu, 20 Aug 2020 21:33:56 +0000 (09:33 +1200)]
Fix a few typos in JIT comments and README
Reviewed-by: Abhijit Menon-Sen
Reviewed-by: Andres Freund
Discussion: https://postgr.es/m/CAApHDvobgmCs6CohqhKTUf7D8vffoZXQTCBTERo9gbOeZmvLTw%40mail.gmail.com
Backpatch-through: 11, where JIT was added
Andres Freund [Thu, 20 Aug 2020 19:59:00 +0000 (12:59 -0700)]
Revert "Make vacuum a bit more verbose to debug BF failure."
This reverts commit
49967da65aec970fcda123acc681f1df5d70bfc6.
Enough time has passed that we can be confident that
07f32fcd23a
resolved the issue. Therefore we can remove the temporary debugging
aids.
Author: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/E1k7tGP-0005V0-5k@gemulon.postgresql.org