Jeff Davis [Sat, 12 Sep 2020 00:10:02 +0000 (17:10 -0700)]
logtape.c: do not preallocate for tapes when sorting
The preallocation logic is only useful for HashAgg, so disable it when
sorting.
Also, adjust an out-of-date comment.
Reviewed-by: Peter Geoghegan
Discussion: https://postgr.es/m/CAH2-Wzn_o7tE2+hRVvwSFghRb75AJ5g-nqGzDUqLYMexjOAe=g@mail.gmail.com
Backpatch-through: 13
Tom Lane [Fri, 11 Sep 2020 20:01:28 +0000 (16:01 -0400)]
Accept SIGQUIT during error recovery in auxiliary processes.
The bgwriter, checkpointer, walwriter, and walreceiver processes
claimed to allow SIGQUIT "at all times". In reality SIGQUIT
would get re-blocked during error recovery, because we didn't
update the actual signal mask immediately, so sigsetjmp() would
save and reinstate a mask that includes SIGQUIT.
This appears to be simply a coding oversight. There's never a
good reason to hold off SIGQUIT in these processes, because it's
going to just call _exit(2) which should be safe enough, especially
since the postmaster is going to tear down shared memory afterwards.
Hence, stick in PG_SETMASK() calls to install the modified BlockSig
mask immediately.
Also try to improve the comments around sigsetjmp blocks. Most of
them were just referencing postgres.c, which is misleading because
actually postgres.c manages the signals differently.
No back-patch, since there's no evidence that this is causing any
problems in the field.
Discussion: https://postgr.es/m/CALDaNm1d1hHPZUg3xU4XjtWBOLCrA+-2cJcLpw-cePZ=GgDVfA@mail.gmail.com
Alvaro Herrera [Fri, 11 Sep 2020 19:15:47 +0000 (16:15 -0300)]
psql: Display stats target of extended statistics
The stats target can be set since commit
d06215d03, but wasn't shown by
psql.
Author: Justin Pryzby <justin@telsasoft.com>
Discussion: https://postgr.es/m/
20200831050047.GG5450@telsasoft.com
Reviewed-by: Georgios Kokolatos <gkokolatos@protonmail.com>
Reviewed-by: Tatsuro Yamada <tatsuro.yamada.tf@nttcom.co.jp>
Tom Lane [Fri, 11 Sep 2020 16:24:46 +0000 (12:24 -0400)]
Log a message when resorting to SIGKILL during shutdown/crash recovery.
Currently, no useful trace is left in the logs when the postmaster
is forced to use SIGKILL to shut down children that failed to respond
to SIGQUIT. Some questions were raised about how often that scenario
happens in the buildfarm, so let's add a LOG-level message showing
that it happened.
Discussion: https://postgr.es/m/
1850884.
1599601164@sss.pgh.pa.us
Tom Lane [Fri, 11 Sep 2020 16:20:16 +0000 (12:20 -0400)]
Don't run atexit callbacks during signal exits from ProcessStartupPacket.
Although
58c6feccf fixed the case for SIGQUIT, we were still calling
proc_exit() from signal handlers for SIGTERM and timeout failures in
ProcessStartupPacket. Fortunately, at the point where that code runs,
we haven't yet connected to shared memory in any meaningful way, so
there is nothing we need to undo in shared memory. This means it
should be safe to use _exit(1) here, ie, not run any atexit handlers
but also inform the postmaster that it's not a crash exit.
To make sure nobody breaks the "nothing to undo" expectation, add
a cross-check that no on-shmem-exit or before-shmem-exit handlers
have been registered yet when we finish using these signal handlers.
This change is simple enough that maybe it could be back-patched,
but I won't risk that right now.
Discussion: https://postgr.es/m/
1850884.
1599601164@sss.pgh.pa.us
Alvaro Herrera [Fri, 11 Sep 2020 15:53:25 +0000 (12:53 -0300)]
Update copyright year
Thinko in
40b3e2c201af.
Reported-by: "Wang, Shenhao" <wangsh.fnst@cn.fujitsu.com>
Discussion: https://postgr.es/m/
ed98706b82694b57a8c0d339a10732aa@G08CNEXMBPEKD06.g08.fujitsu.local
Amit Kapila [Fri, 11 Sep 2020 04:30:01 +0000 (10:00 +0530)]
Skip empty transaction stream in test_decoding.
We were decoding empty transactions via streaming APIs added in commit
45fdc9738b even when the user used the option 'skip-empty-xacts'. The APIs
makes no effort to skip empty xacts under the assumption that we will
never try to stream such transactions. However, that is not true because
we can pick to stream a transaction that has change messages for
REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT and we don't send such messages to
downstream rather they are just to update the internal state. So, we need
to skip such xacts when plugin uses the option 'skip-empty-xacts'.
Diagnosed-By: Amit Kapila
Author: Dilip Kumar
Reviewed-by: Amit Kapila
Discussion: https://postgr.es/m/CAA4eK1+OqgFNZkf7=ETe_y5ntjgDk3T0wcdkd4Sot_u1hySGfw@mail.gmail.com
Alvaro Herrera [Thu, 10 Sep 2020 22:37:02 +0000 (19:37 -0300)]
Print WAL logical message contents in pg_waldump
This helps debuggability when looking at WAL streams containing logical
messages.
Author: Ashutosh Bapat <ashutosh.bapat@2ndquadrant.com>
Reviewed-by: Álvaro Herrera <alvherre@alvh.no-ip.org>
Discussion: https://postgr.es/m/CAExHW5sWx49rKmXbg5H1Xc1t+nRv9PaYKQmgw82HPt6vWDVmDg@mail.gmail.com
Tom Lane [Thu, 10 Sep 2020 16:06:26 +0000 (12:06 -0400)]
Use _exit(2) for SIGQUIT during ProcessStartupPacket, too.
Bring the signal handling for startup-packet collection into line
with the policy established in commits
bedadc732 and
8e19a8264,
namely don't risk running atexit callbacks when handling SIGQUIT.
Ideally, we'd not do so for SIGTERM or timeout interrupts either,
but that change seems a bit too risky for the back branches.
For now, just improve the comments in this area to describe the risk.
Also relocate where BackendInitialize re-disables these interrupts,
to minimize the code span where they're active. This doesn't buy
a whole lot of safety, but it can't hurt.
In passing, rename startup_die() to remove confusion about whether
it is for the startup process.
Like the previous commits, back-patch to all supported branches.
Discussion: https://postgr.es/m/
1850884.
1599601164@sss.pgh.pa.us
Robert Haas [Thu, 10 Sep 2020 15:10:55 +0000 (11:10 -0400)]
New contrib module, pg_surgery, with heap surgery functions.
Sometimes it happens that the visibility information for a tuple
becomes corrupted, either due to bugs in the database software or
external factors. Provide a function heap_force_kill() that can
be used to truncate such dead tuples to dead line pointers, and
a function heap_force_freeze() that can be used to overwrite the
visibility information in such a way that the tuple becomes
all-visible.
These functions are unsafe, in that you can easily use them to
corrupt a database that was not previously corrupted, and you can
use them to further corrupt an already-corrupted database or to
destroy data. The documentation accordingly cautions against
casual use. However, in some cases they permit recovery of data
that would otherwise be very difficult to recover, or to allow a
system to continue to function when it would otherwise be difficult
to do so.
Because we may want to add other functions for performing other
kinds of surgery in the future, the new contrib module is called
pg_surgery rather than something specific to these functions. I
proposed back-patching this so that it could be more easily used
by people running existing releases who are facing these kinds of
problems, but that proposal did not attract enough support, so
no back-patch for now.
Ashutosh Sharma, reviewed and tested by Andrey M. Borodin,
M. Beena Emerson, Masahiko Sawada, Rajkumar Raghuwanshi,
Asim Praveen, and Mark Dilger, and somewhat revised by me.
Discussion: http://postgr.es/m/CA+TgmoZW1fsU-QUNCRUQMGUygBDPVeOTLCqRdQZch=EYZnctSA@mail.gmail.com
Peter Eisentraut [Thu, 10 Sep 2020 14:13:19 +0000 (16:13 +0200)]
Remove unused parameter
Apparently, this was never used when
introduced (
3dad73e71f08abd86564d5090a58ca71740e07e0).
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Peter Eisentraut [Thu, 10 Sep 2020 13:55:31 +0000 (15:55 +0200)]
Add libpq's openssl dependencies to pkg-config file
Add libssl and libcrypto to libpq.pc's Requires.private. This allows
static linking to work if those libssl or libcrypto themselves have
dependencies in their *.private fields, such as -lz in some cases.
Reported-by: Sandro Mani <manisandro@gmail.com>
Discussion: https://www.postgresql.org/message-id/flat/
837d1dcf-2fca-ee6e-0d7e-
6bce1a1bac75@gmail.com
Peter Eisentraut [Thu, 10 Sep 2020 13:31:09 +0000 (15:31 +0200)]
doc: Remove buggy ICU collation from documentation
We have had multiple reports that point to the
'@colReorder=latn-digit' collation customization being buggy. We have
reported this to ICU and are waiting for a fix. In the meantime,
remove references to this from the documentation and replace it by
another reordering example. Apparently, many users have been picking
up this example specifically from the documentation.
Author: Jehan-Guillaume de Rorthais <jgdr@dalibo.com>
Discussion: https://www.postgresql.org/message-id/flat/
153201618542.1404.
3611626898935613264%40wrigleys.postgresql.org
Peter Eisentraut [Thu, 10 Sep 2020 12:52:36 +0000 (14:52 +0200)]
Add more tests for EXTRACT of date type
EXTRACT of date type is implemented as a wrapper around EXTRACT of
timestamp, so the code is already tested there. But the externally
visible behavior of EXTRACT on date is not recorded anywhere. Since
there is some discussion about reimplementing or refactoring some of
this, add some more explicit tests of EXTRACT on date, similar in
structure to existing EXTRACT tests on other data types.
Discussion: https://www.postgresql.org/message-id/flat/
42b73d2d-da12-ba9f-570a-
420e0cce19d9@phystech.edu
Magnus Hagander [Thu, 10 Sep 2020 12:15:26 +0000 (14:15 +0200)]
Fix title in reference section
Reported-by: Robert Kahlert
Author: Daniel Gustafsson
Etsuro Fujita [Thu, 10 Sep 2020 09:00:00 +0000 (18:00 +0900)]
Clean up some code and comments in partbounds.c.
Do some minor cleanup for commit
c8434d64c: 1) remove a useless
assignment (in normal builds) and 2) improve comments a little.
Back-patch to v13 where the aforementioned commit went in.
Author: Etsuro Fujita
Reviewed-by: Alvaro Herrera
Discussion: https://postgr.es/m/CAPmGK16yCd2R4=bQ4g8N2dT9TtA5ZU+qNmJ3LPc_nypbNy4_2A@mail.gmail.com
Michael Paquier [Thu, 10 Sep 2020 06:50:19 +0000 (15:50 +0900)]
doc: Fix some grammar and inconsistencies
Some comments are fixed while on it.
Author: Justin Pryzby
Discussion: https://postgr.es/m/
20200818171702.GK17022@telsasoft.com
Backpatch-through: 9.6
Noah Misch [Thu, 10 Sep 2020 01:50:24 +0000 (18:50 -0700)]
Fix rd_firstRelfilenodeSubid for nailed relations, in parallel workers.
Move applicable code out of RelationBuildDesc(), which nailed relations
bypass. Non-assert builds experienced no known problems. Back-patch to
v13, where commit
c6b92041d38512a4176ed76ad06f713d2e6c01a8 introduced
rd_firstRelfilenodeSubid.
Kyotaro Horiguchi. Reported by Justin Pryzby.
Discussion: https://postgr.es/m/
20200907023737.GA7158@telsasoft.com
Tom Lane [Wed, 9 Sep 2020 19:32:34 +0000 (15:32 -0400)]
Make archiver's SIGQUIT handler exit via _exit().
Commit
8e19a8264 changed the SIGQUIT handlers of almost all server
processes not to run atexit callbacks. The archiver process was
skipped, perhaps because it's not connected to shared memory; but
it's just as true here that running atexit callbacks in a signal
handler is unsafe. So let's make it work like the rest.
In HEAD and v13, we can use the common SignalHandlerForCrashExit
handler. Before that, just tweak pgarch_exit to use _exit(2)
explicitly.
Like the previous commit, back-patch to all supported branches.
Kyotaro Horiguchi, back-patching by me
Discussion: https://postgr.es/m/
1850884.
1599601164@sss.pgh.pa.us
Peter Eisentraut [Wed, 9 Sep 2020 18:16:28 +0000 (20:16 +0200)]
Expose internal function for converting int64 to numeric
Existing callers had to take complicated detours via
DirectFunctionCall1(). This simplifies a lot of code.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://www.postgresql.org/message-id/flat/
42b73d2d-da12-ba9f-570a-
420e0cce19d9@phystech.edu
Tom Lane [Wed, 9 Sep 2020 16:00:49 +0000 (12:00 -0400)]
Doc: adjust documentation related to index support functions.
Commit
15cb2bd27 neglected to make the running text match the
tables, leaving the reader with the strong impression that
we cannot count. Also, don't drop an unrelated para between
a table and the para describing it.
Tom Lane [Wed, 9 Sep 2020 15:53:39 +0000 (11:53 -0400)]
Minor fixes in docs and error messages.
Alexander Lakhin
Discussion: https://postgr.es/m/
ce7debdd-c943-d7a7-9b41-
687107b27831@gmail.com
Magnus Hagander [Wed, 9 Sep 2020 10:20:53 +0000 (12:20 +0200)]
Add missing quote in docs
Mistake in commit
68b603e1a9.
Reported-by: Ian Barwick
Peter Eisentraut [Wed, 9 Sep 2020 07:58:12 +0000 (09:58 +0200)]
Add some more numeric test coverage
max(numeric) wasn't tested at all, min(numeric) was only used by some
unrelated tests. Add explicit tests with the other numeric aggregate
functions.
Alvaro Herrera [Tue, 8 Sep 2020 22:35:15 +0000 (19:35 -0300)]
Check default partitions constraints while descending
Partitioning tuple route code assumes that the partition chosen while
descending the partition hierarchy is always the correct one. This is
true except when the partition is the default partition and another
partition has been added concurrently: the partition constraint changes
and we don't recheck it. This can lead to tuples mistakenly being added
to the default partition that should have been rejected.
Fix by rechecking the default partition constraint while descending the
hierarchy.
An isolation test based on the reproduction steps described by Hao Wu
(with tweaks for extra coverage) is included.
Backpatch to 12, where this bug came in with
898e5e3290a7.
Reported by: Hao Wu <hawu@vmware.com>
Author: Amit Langote <amitlangote09@gmail.com>
Author: Álvaro Herrera <alvherre@alvh.no-ip.org>
Discussion: https://postgr.es/m/CA+HiwqFqBmcSSap4sFnCBUEL_VfOMmEKaQ3gwUhyfa4c7J_-nA@mail.gmail.com
Discussion: https://postgr.es/m/DM5PR0501MB3910E97A9EDFB4C775CF3D75A42F0@DM5PR0501MB3910.namprd05.prod.outlook.com
Tom Lane [Tue, 8 Sep 2020 19:54:25 +0000 (15:54 -0400)]
Install an error check into cancel_before_shmem_exit().
Historically, cancel_before_shmem_exit() just silently did nothing
if the specified callback wasn't the top-of-stack. The folly of
ignoring this case was exposed by the bugs fixed in
303640199 and
bab150045, so let's make it throw elog(ERROR) instead.
There is a decent argument to be made that PG_ENSURE_ERROR_CLEANUP
should use some separate infrastructure, so it wouldn't break if
something inside the guarded code decides to register a new
before_shmem_exit callback. However, a survey of the surviving
uses of before_shmem_exit() and PG_ENSURE_ERROR_CLEANUP doesn't
show any plausible conflicts of that sort today, so for now we'll
forgo the extra complexity. (It will almost certainly become
necessary if anyone ever wants to wrap PG_ENSURE_ERROR_CLEANUP
around arbitrary user-defined actions, though.)
No backpatch, since this is developer support not a production issue.
Bharath Rupireddy, per advice from Andres Freund, Robert Haas, and myself
Discussion: https://postgr.es/m/CALj2ACWk7j4F2v2fxxYfrroOF=AdFNPr1WsV+AGtHAFQOqm_pw@mail.gmail.com
Andres Freund [Tue, 8 Sep 2020 18:25:34 +0000 (11:25 -0700)]
Fix autovacuum cancellation.
The problem is caused by me (Andres) having ProcSleep() look at the
wrong PGPROC entry in
5788e258bb2.
Unfortunately it seems hard to write a reliable test for autovacuum
cancellations. Perhaps somebody will come up with a good approach, but
it seems worth fixing the issue even without a test.
Reported-By: Jeff Janes <jeff.janes@gmail.com>
Author: Jeff Janes <jeff.janes@gmail.com>
Discussion: https://postgr.es/m/CAMkU=1wH2aUy+wDRDz+5RZALdcUnEofV1t9PzXS_gBJO9vZZ0Q@mail.gmail.com
Tom Lane [Tue, 8 Sep 2020 15:47:37 +0000 (11:47 -0400)]
Use plain memset() in numeric.c, not MemSet and friends.
This essentially reverts a micro-optimization I made years ago,
as part of the much larger commit
d72f6c750. It's doubtful
that there was any hard evidence for it being helpful even then,
and the case is even more dubious now that modern compilers
are so much smarter about inlining memset().
The proximate reason for undoing it is to get rid of the type punning
inherent in MemSet, for fear that that may cause problems now that
we're applying additional optimization switches to numeric.c.
At the very least this'll silence some warnings from a few old
buildfarm animals.
(It's probably past time for another look at whether MemSet is still
worth anything at all, but I do not propose to tackle that question
right now.)
Discussion: https://postgr.es/m/CAJ3gD9evtA_vBo+WMYMyT-u=keHX7-r8p2w7OSRfXf42LTwCZQ@mail.gmail.com
Peter Eisentraut [Tue, 8 Sep 2020 15:11:16 +0000 (17:11 +0200)]
Use <unnamed> for name of unnamed portal's memory context
Otherwise just printing an empty string makes the memory context debug
output slightly confusing.
Discussion: https://www.postgresql.org/message-id/flat/
ccb353ef-89ff-09b3-8046-
1d2514624b9c%402ndquadrant.com
Peter Eisentraut [Tue, 8 Sep 2020 08:08:46 +0000 (10:08 +0200)]
Remove unused parameter
unused since
f0d6f20278b7c5c412ce40a9b86c6b31dc2fbfdd
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Michael Paquier [Tue, 8 Sep 2020 02:15:21 +0000 (11:15 +0900)]
Remove isolation test reindex-partitions
The isolation test added by
a6642b3 is proving to be unstable, as once
the first transaction holding a lock on the top-most partitioned table
or on a partition commits, the commit order of the follow-up DROP TABLE
and REINDEX could become reversed depending on the timing.
The only part of the test that could be entirely reliable is the one
using a SHARE lock, allowing REINDEX to commit first, but it is the
least interesting of the set.
Per buildfarm members rorqual and mylodon.
Discussion: https://postgr.es/m/E1kFSBj-00062c-Mu@gemulon.postgresql.org
Michael Paquier [Tue, 8 Sep 2020 01:09:22 +0000 (10:09 +0900)]
Add support for partitioned tables and indexes in REINDEX
Until now, REINDEX was not able to work with partitioned tables and
indexes, forcing users to reindex partitions one by one. This extends
REINDEX INDEX and REINDEX TABLE so as they can accept a partitioned
index and table in input, respectively, to reindex all the partitions
assigned to them with physical storage (foreign tables, partitioned
tables and indexes are then discarded).
This shares some logic with schema and database REINDEX as each
partition gets processed in its own transaction after building a list of
relations to work on. This choice has the advantage to minimize the
number of invalid indexes to one partition with REINDEX CONCURRENTLY in
the event a cancellation or failure in-flight, as the only indexes
handled at once in a single REINDEX CONCURRENTLY loop are the ones from
the partition being working on.
Isolation tests are added to emulate some cases I bumped into while
developing this feature, particularly with the concurrent drop of a
leaf partition reindexed. However, this is rather limited as LOCK would
cause REINDEX to block in the first transaction building the list of
partitions.
Per its multi-transaction nature, this new flavor cannot run in a
transaction block, similarly to REINDEX SCHEMA, SYSTEM and DATABASE.
Author: Justin Pryzby, Michael Paquier
Reviewed-by: Anastasia Lubennikova
Discussion: https://postgr.es/m/
db12e897-73ff-467e-94cb-
4af03705435f.adger.lj@alibaba-inc.com
Jeff Davis [Mon, 7 Sep 2020 20:31:59 +0000 (13:31 -0700)]
Adjust cost model for HashAgg that spills to disk.
Tomas Vondra observed that the IO behavior for HashAgg tends to be
worse than for Sort. Penalize HashAgg IO costs accordingly.
Also, account for the CPU effort of spilling the tuples and reading
them back.
Discussion: https://postgr.es/m/
20200906212112.nzoy5ytrzjjodpfh@development
Reviewed-by: Tomas Vondra
Backpatch-through: 13
Tom Lane [Mon, 7 Sep 2020 18:52:33 +0000 (14:52 -0400)]
Clarify comments in enforce_generic_type_consistency().
Some of the pre-existing comments were vague about whether they
referred to all polymorphic types or only the old-style ones.
Also be more consistent about using the "family 1" vs "family 2"
terminology.
Himanshu Upadhyaya and Tom Lane
Discussion: https://postgr.es/m/CAPF61jBUg9XoMPNuLpoZ+h6UZ2VxKdNt3rQL1xw1GOBwjWzAXQ@mail.gmail.com
Tom Lane [Mon, 7 Sep 2020 16:03:04 +0000 (12:03 -0400)]
Frob numeric.c loop so that clang will auto-vectorize it too.
Experimentation shows that clang will auto-vectorize the critical
multiplication loop if the termination condition is written "i2 < limit"
rather than "i2 <= limit". This seems unbelievably stupid, but I've
reproduced it on both clang 9.0.1 (RHEL8) and 11.0.3 (macOS Catalina).
gcc doesn't care, so tweak the code to do it that way.
Discussion: https://postgr.es/m/CAJ3gD9evtA_vBo+WMYMyT-u=keHX7-r8p2w7OSRfXf42LTwCZQ@mail.gmail.com
Thomas Munro [Mon, 7 Sep 2020 11:18:22 +0000 (23:18 +1200)]
Add d_type to our Windows dirent emulation.
This allows us to skip some stat calls, by extending commit
861c6e7c to
cover Windows systems.
Author: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>
Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Magnus Hagander <magnus@hagander.net>
Reviewed-by: Thomas Munro <thomas.munro@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKG%2BFzxupGGN4GpUdbzZN%2Btn6FQPHo8w0Q%2BAPH5Wz8RG%2Bww%40mail.gmail.com
Peter Eisentraut [Mon, 7 Sep 2020 07:59:50 +0000 (09:59 +0200)]
doc: Fix table cell overflow
Fix one instance of a table cell overflow by adding a zero-width
space. The visual impact of this is minimal, but since this is
currently the only such case reported by FOP ("contents of ... exceed
the available area"), it seems worth getting rid of.
Thomas Munro [Mon, 7 Sep 2020 06:11:46 +0000 (18:11 +1200)]
Skip unnecessary stat() calls in walkdir().
Some kernels can tell us the type of a "dirent", so we can avoid a call
to stat() or lstat() in many cases. Define a new function
get_dirent_type() to contain that logic, for use by the backend and
frontend versions of walkdir(), and perhaps other callers in future.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Reviewed-by: Juan José Santamaría Flecha <juanjo.santamaria@gmail.com>
Discussion: https://postgr.es/m/CA%2BhUKG%2BFzxupGGN4GpUdbzZN%2Btn6FQPHo8w0Q%2BAPH5Wz8RG%2Bww%40mail.gmail.com
Michael Paquier [Mon, 7 Sep 2020 05:34:59 +0000 (14:34 +0900)]
doc: Tweak sentence for pg_checksums when enabling checksums
The previous version of the docs mentioned that files are rewritten,
implying that a second copy of each file gets created, but each file is
updated in-place.
Author: Michael Banck
Reviewed-by: Daniel Gustafsson, Michael Paquier
Discussion: https://postgr.es/m/
858086b6a42fb7d17995b6175856f7e7ec44d0a2.camel@credativ.de
Backpatch-through: 12
Amit Kapila [Mon, 7 Sep 2020 02:38:58 +0000 (08:08 +0530)]
Add additional tests to test streaming of in-progress transactions.
This covers the functionality tests for streaming in-progress
subtransactions, streaming transactions containing rollback to savepoints,
and streaming transactions having DDLs.
Author: Tomas Vondra, Amit Kapila and Dilip Kumar
Reviewed-by: Dilip Kumar
Discussion: https://postgr.es/m/
688b0b7f-2f6c-d827-c27b-
216a8e3ea700@2ndquadrant.com
Tom Lane [Mon, 7 Sep 2020 01:40:39 +0000 (21:40 -0400)]
Apply auto-vectorization to the inner loop of numeric multiplication.
Compile numeric.c with -ftree-vectorize where available, and adjust
the innermost loop of mul_var() so that it is amenable to being
auto-vectorized. (Mainly, that involves making it process the arrays
left-to-right not right-to-left.)
Applying -ftree-vectorize actually makes numeric.o smaller, at least
with my compiler (gcc 8.3.1 on x86_64), and it's a little faster too.
Independently of that, fixing the inner loop to be vectorizable also
makes things a bit faster. But doing both is a huge win for
multiplications with lots of digits. For me, the numeric regression
test is the same speed to within measurement noise, but numeric_big
is a full 45% faster.
We also looked into applying -funroll-loops, but that makes numeric.o
bloat quite a bit, and the additional speed improvement is very
marginal.
Amit Khandekar, reviewed and edited a little by me
Discussion: https://postgr.es/m/CAJ3gD9evtA_vBo+WMYMyT-u=keHX7-r8p2w7OSRfXf42LTwCZQ@mail.gmail.com
Tom Lane [Mon, 7 Sep 2020 01:28:16 +0000 (21:28 -0400)]
Split Makefile symbol CFLAGS_VECTOR into two symbols.
Replace CFLAGS_VECTOR with CFLAGS_UNROLL_LOOPS and CFLAGS_VECTORIZE,
allowing us to distinguish whether we want to apply -funroll-loops,
-ftree-vectorize, or both to a particular source file. Up to now
the only consumer of the symbol has been checksum.c which wants
both, so that there was no need to distinguish; but that's about
to change.
Amit Khandekar, reviewed and edited a little by me
Discussion: https://postgr.es/m/CAJ3gD9evtA_vBo+WMYMyT-u=keHX7-r8p2w7OSRfXf42LTwCZQ@mail.gmail.com
Tom Lane [Sun, 6 Sep 2020 18:13:02 +0000 (14:13 -0400)]
Remove arbitrary line length limits in pg_regress (plain and ECPG).
Refactor replace_string() to use a StringInfo for the modifiable
string argument. This allows the string to be of indefinite size
initially and/or grow substantially during replacement. The previous
logic in convert_sourcefiles_in() had a hard-wired limit of 1024
bytes on any line in input/*.sql or output/*.out files. While we've
not had reports of trouble yet, it'd surely have bit us someday.
This also fixes replace_string() so it won't get into an infinite
loop if the string-to-be-replaced is a substring of the replacement.
That's unlikely to happen in current usage, but the function surely
shouldn't depend on it.
Also fix ecpg_filter() to use a StringInfo and thereby remove its
hard limit of 300 bytes on the length of an ecpg source line.
Asim Rama Praveen and Georgios Kokolatos,
reviewed by Alvaro Herrera and myself
Discussion: https://postgr.es/m/y9Dlk2QhiZ39DhaB1QE9mgZ95HcOQKZCNtGwN7XCRKMdBRBnX_0woaRUtTjloEp4PKA6ERmcUcfq3lPGfKPOJ5xX2TV-5WoRYyySeNHRzdw=@protonmail.com
Tom Lane [Sun, 6 Sep 2020 17:57:10 +0000 (13:57 -0400)]
Refactor pg_get_line() to expose an alternative StringInfo-based API.
Letting the caller provide a StringInfo to read into is helpful when
the caller needs to merge lines or otherwise modify the data after
it's been read. Notably, now the code added by commit
8f8154a50
can use pg_get_line_append() instead of having its own copy of that
logic. A follow-on commit will also make use of this.
Also, since StringInfo buffers are a minimum of 1KB long, blindly
using pg_get_line() in a loop can eat a lot more memory than one would
expect. I discovered for instance that commit
e0f05cd5b caused initdb
to consume circa 10MB to read postgres.bki, even though that's under
1MB worth of data. A less memory-hungry alternative is to re-use the
same StringInfo for all lines and pg_strdup the results.
Discussion: https://postgr.es/m/
1315832.
1599345736@sss.pgh.pa.us
Magnus Hagander [Sun, 6 Sep 2020 17:28:32 +0000 (19:28 +0200)]
Change path in example of file_fdw for logs
It's better to use a relative path into the data directory, than to a
hardcoded home directory of user 'josh'.
Discussion: https://postgr.es/m/CABUevEyuf67Yu_r9gpDMs5MKifK7+-+pe=ZjKzya4JEn9kUk1w@mail.gmail.com
Magnus Hagander [Sun, 6 Sep 2020 17:26:55 +0000 (19:26 +0200)]
Fix typo in comment
Author: Hou, Zhijie
Tom Lane [Sun, 6 Sep 2020 16:55:13 +0000 (12:55 -0400)]
Fix misleading error message about inconsistent moving-aggregate types.
We reported the wrong types when complaining that an aggregate's
moving-aggregate implementation is inconsistent with its regular
implementation.
This was wrong since the feature was introduced, so back-patch
to all supported branches.
Jeff Janes
Discussion: https://postgr.es/m/CAMkU=1x808LH=LPhZp9mNSP0Xd1xDqEd+XeGcvEe48dfE6xV=A@mail.gmail.com
Tom Lane [Sun, 6 Sep 2020 15:50:40 +0000 (11:50 -0400)]
Remove useless lstat() call in pg_rewind.
This is duplicative of an lstat that was just done by the calling
function (traverse_datadir), besides which we weren't really doing
anything with the results. There's not much point in checking to
see if someone removed the file since the previous lstat, since the
FILE_ACTION_REMOVE code would have to deal with missing-file cases
anyway. Moreover, the "exists = false" assignment was a dead store;
nothing was done with that value later.
A syscall saved is a syscall earned, so back-patch to 9.5
where this code was introduced.
Discussion: https://postgr.es/m/
1221796.
1599329320@sss.pgh.pa.us
Peter Eisentraut [Sun, 6 Sep 2020 14:46:13 +0000 (16:46 +0200)]
doc: Don't hide the "Up" link when it is the same as "Home"
The original stylesheets seemed to think this was a good idea, but our
users find it confusing and unhelpful, so undo that logic.
Reported-by: Fabien COELHO <coelho@cri.ensmp.fr>
Discussion: https://www.postgresql.org/message-id/flat/alpine.DEB.2.22.394.
2006210914370.859381%40pseudo
Peter Eisentraut [Sun, 6 Sep 2020 07:32:16 +0000 (09:32 +0200)]
Remove unused parameter
unused since
84d723b6cefcf25b8c800f8aa6cf3c9538a546b4
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Tom Lane [Sat, 5 Sep 2020 20:20:04 +0000 (16:20 -0400)]
Improve some ancient, crufty code in bootstrap + initdb.
At some point back in the last century, somebody felt that reading
all of pg_type twice was cheaper, or at least easier, than using
repalloc() to resize the Typ[] array dynamically. That seems like an
entirely wacko proposition, so rewrite the code to do it the other
way. (To add insult to injury, there were two not-quite-identical
copies of said code.)
initdb.c's readfile() function had the same disease of preferring
to do double the I/O to avoid resizing its output array. Here,
we can make things easier by using the just-invented pg_get_line()
function to handle reading individual lines without a predetermined
notion of how long they are.
On my machine, it's difficult to detect any net change in the
overall runtime of initdb from these changes; but they should
help on slower buildfarm machines (especially since a buildfarm
cycle involves a lot of initdb's these days).
My attention was drawn to these places by scan-build complaints,
but on inspection they needed a lot more work than just suppressing
dead stores :-(
Tom Lane [Sat, 5 Sep 2020 17:17:32 +0000 (13:17 -0400)]
Yet more elimination of dead stores and useless initializations.
I'm not sure what tool Ranier was using, but the ones I contributed
were found by using a newer version of scan-build than I tried before.
Ranier Vilela and Tom Lane
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Michael Paquier [Sat, 5 Sep 2020 12:33:53 +0000 (21:33 +0900)]
Switch to multi-inserts when registering dependencies for many code paths
This commit improves the dependency registrations by taking advantage of
the preliminary work done in
63110c62, to group together the insertion
of dependencies of the same type to pg_depend. With the current layer
of routines available, and as only dependencies of the same type can be
grouped, there are code paths still doing more than one multi-insert
when it is necessary to register dependencies of multiple types
(constraint and index creation are two cases doing that).
While on it, this refactors some of the code to use ObjectAddressSet()
when manipulating object addresses.
Author: Daniel Gustafsson, Michael Paquier
Reviewed-by: Andres Freund, Álvaro Herrera
Discussion: https://postgr.es/m/
20200807061619.GA23955@paquier.xyz
Peter Eisentraut [Sat, 5 Sep 2020 11:28:05 +0000 (13:28 +0200)]
Extend SQL function tests lightly
The basic tests that defined SQL functions didn't actually run the
functions to see if they worked. Add that, and also fix a minor
mistake in a function that was revealed by this. (This is not a
question of test coverage, since there are other places where SQL
functions are run, but it is a bit of a silly test design.)
Discussion: https://www.postgresql.org/message-id/flat/
1c11f1eb-f00c-43b7-799d-
2d44132c02d7@2ndquadrant.com
Peter Eisentraut [Sat, 5 Sep 2020 09:32:20 +0000 (11:32 +0200)]
Fix typo in comment
Michael Paquier [Sat, 5 Sep 2020 04:52:47 +0000 (13:52 +0900)]
Use multi-inserts for pg_depend
This is a follow-up of the work done in
e3931d01. This case is a bit
different than pg_attribute and pg_shdepend: the maximum number of items
to insert is known in advance, but there is no need to handle pinned
dependencies. Hence, the base allocation for slots is done based on the
number of items and the maximum allowed with a cap at 64kB. Slots are
initialized once used to minimize the overhead of the operation.
The insertions can be done for dependencies of the same type. More
could be done by grouping the insertion of multiple dependency types in
a single batch. This is left as future work.
Some of the multi-insert logic is also simplified for pg_shdepend, as
per the feedback discussed for this specific patch. This also moves to
indexing.h the variable capping the maximum amount of data that can be
used at once for a multi-insert, instead of having separate definitions
for pg_attribute, pg_depend and pg_shdepend.
Author: Daniel Gustafsson, Michael Paquier
Reviewed-by: Andres Freund, Álvaro Herrera
Discussion: https://postgr.es/m/
20200807061619.GA23955@paquier.xyz
Tom Lane [Sat, 5 Sep 2020 01:01:58 +0000 (21:01 -0400)]
Make new authentication test case more robust.
I happened to notice that the new test case I added in
b55b4dad9
falls over if one runs "make check" repeatedly; though not in branches
after v10. That's because it was assuming that tmp_check/pgpass
wouldn't exist already. However, it's only been since v11 that the
Makefiles forcibly remove all of tmp_check/ before starting a TAP run.
This fix to unlink the file is therefore strictly necessary only in
v10 ... but it seems wisest to do it across the board, rather than
let the test rely on external logic to get the conditions right.
Tom Lane [Sat, 5 Sep 2020 00:20:05 +0000 (20:20 -0400)]
Fix over-eager ping'ing in logical replication receiver.
Commit
3f60f690f only partially fixed the broken-status-tracking
issue in LogicalRepApplyLoop: we need ping_sent to have the same
lifetime as last_recv_timestamp. The effects are much less serious
than what that commit fixed, though. AFAICS this would just lead to
extra ping requests being sent, once per second until the sender
responds. Still, it's a bug, so backpatch to v10 as before.
Discussion: https://postgr.es/m/959627.
1599248476@sss.pgh.pa.us
Tom Lane [Fri, 4 Sep 2020 22:17:47 +0000 (18:17 -0400)]
Remove still more useless assignments.
Fix some more things scan-build pointed to as dead stores. In some of
these cases, rearranging the code a little leads to more readable
code IMO. It's all cosmetic, though.
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Jeff Davis [Fri, 4 Sep 2020 19:01:58 +0000 (12:01 -0700)]
Fix bogus MaxAllocSize check in logtape.c.
Reported-by: Peter Geoghegan
Discussion: https://postgr.es/m/CAH2-Wz=NZPZc3-fkdmvu=w2itx0PiB-G6QpxHXZOjuvFAzPdZw@mail.gmail.com
Backpatch-through: 13
Alvaro Herrera [Fri, 4 Sep 2020 18:58:32 +0000 (14:58 -0400)]
Report expected contrecord length on mismatch
When reading a WAL record fails to find continuation record(s) of the
proper length, report what it expects, for clarity.
Reviewed-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/
20200903212152.GA15319@alvherre.pgsql
Tom Lane [Fri, 4 Sep 2020 18:32:10 +0000 (14:32 -0400)]
Remove some more useless assignments.
Found with clang's scan-build tool. It also whines about a lot of
other dead stores that we should *not* change IMO, either as a matter
of style or future-proofing. But these places seem like clear
oversights.
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Andrew Dunstan [Fri, 4 Sep 2020 17:53:09 +0000 (13:53 -0400)]
Collect attribute data on extension owned tables being dumped
If this data is not collected, pg_dump segfaults if asked for column
inserts.
Fix by Fabrízio de Royes Mello
Backpatch to release 12 where the bug was introduced.
Bruce Momjian [Fri, 4 Sep 2020 17:27:52 +0000 (13:27 -0400)]
C comment: correct use of 64-"byte" cache line size
Reported-by: Kelly Min
Discussion: https://postgr.es/m/CAPSbxatOiQO90LYpSC3+svAU9-sHgDfEP4oFhcEUt_X=DqFA9g@mail.gmail.com
Backpatch-through: 9.5
Tom Lane [Fri, 4 Sep 2020 16:40:28 +0000 (12:40 -0400)]
Fix rare deadlock failure in create_am regression test.
The "DROP ACCESS METHOD gist2" test will require locking the index
to be dropped and then its table; while most ordinary operations
lock a table first then its index. While no concurrent test scripts
should be touching fast_emp4000, autovacuum might chance to be
processing that table when the DROP runs, resulting in a deadlock
failure. This is pretty rare but we see it in the buildfarm from
time to time.
To fix, acquire a lock on fast_emp4000 before issuing the DROP.
Since the point of the exercise is mostly to prevent buildfarm
failures, back-patch to 9.6 where this test was introduced.
Discussion: https://postgr.es/m/839004.
1599185607@sss.pgh.pa.us
Peter Eisentraut [Fri, 4 Sep 2020 06:45:57 +0000 (08:45 +0200)]
doc: Change table alias names to lower case in tutorial chapter
This is needlessly different from our usual style otherwise.
Author: Jürgen Purtz <juergen@purtz.de>
Discussion: https://www.postgresql.org/message-id/flat/
158996922318.7035.
10603922579567326239@wrigleys.postgresql.org
Peter Eisentraut [Fri, 4 Sep 2020 06:39:01 +0000 (08:39 +0200)]
doc: Fix whitespace issue in PDF
Move <indexterm> outside of <para> to avoid whitespace issue in PDF
output.
Author: Jürgen Purtz <juergen@purtz.de>
Discussion: https://www.postgresql.org/message-id/flat/
158996922318.7035.
10603922579567326239@wrigleys.postgresql.org
Peter Eisentraut [Fri, 4 Sep 2020 06:19:20 +0000 (08:19 +0200)]
doc: Use tags consistently in the tutorial chapter
Make more consistent use of <screen> and <programlisting>.
Author: Jürgen Purtz <juergen@purtz.de>
Discussion: https://www.postgresql.org/message-id/flat/
158996922318.7035.
10603922579567326239@wrigleys.postgresql.org
Peter Eisentraut [Fri, 4 Sep 2020 06:02:58 +0000 (08:02 +0200)]
Remove unused parameter
unused since
93ee38eade1b2b4964354b95b01b09e17d6f098d
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Amit Kapila [Fri, 4 Sep 2020 05:55:16 +0000 (11:25 +0530)]
Fix inline marking introduced in commit
464824323e.
Forgot to add inline marking in changes_filename() declaration. In the passing, add
inline marking for a similar function subxact_filename().
Reported-By: Nathan Bossart
Discussion: https://postgr.es/m/
E98FBE8F-B878-480D-A728-
A60C6EED3047@amazon.com
Bruce Momjian [Fri, 4 Sep 2020 02:57:35 +0000 (22:57 -0400)]
remove redundant initializations
Reported-by: Ranier Vilela
Discussion: https://postgr.es/m/CAEudQAo1+AcGppxDSg8k+zF4+Kv+eJyqzEDdbpDg58-=MQcerQ@mail.gmail.com
Author: Ranier Vilela
Backpatch-through: master
Michael Paquier [Fri, 4 Sep 2020 01:36:35 +0000 (10:36 +0900)]
Remove variable "concurrent" from ReindexStmt
This node already handles multiple options using a bitmask, so having a
separate boolean flag is not necessary. This simplifies the code a bit
with less arguments to give to the reindex routines, by replacing the
boolean with an equivalent bitmask value.
Reviewed-by: Julien Rouhaud
Discussion: https://postgr.es/m/
20200902110326.GA14963@paquier.xyz
Tom Lane [Fri, 4 Sep 2020 00:09:18 +0000 (20:09 -0400)]
Remove arbitrary restrictions on password length.
This patch started out with the goal of harmonizing various arbitrary
limits on password length, but after awhile a better idea emerged:
let's just get rid of those fixed limits.
recv_password_packet() has an arbitrary limit on the packet size,
which we don't really need, so just drop it. (Note that this doesn't
really affect anything for MD5 or SCRAM password verification, since
those will hash the user's password to something shorter anyway.
It does matter for auth methods that require a cleartext password.)
Likewise remove the arbitrary error condition in pg_saslprep().
The remaining limits are mostly in client-side code that prompts
for passwords. To improve those, refactor simple_prompt() so that
it allocates its own result buffer that can be made as big as
necessary. Actually, it proves best to make a separate routine
pg_get_line() that has essentially the semantics of fgets(), except
that it allocates a suitable result buffer and hence will never
return a truncated line. (pg_get_line has a lot of potential
applications to replace randomly-sized fgets buffers elsewhere,
but I'll leave that for another patch.)
I built pg_get_line() atop stringinfo.c, which requires moving
that code to src/common/; but that seems fine since it was a poor
fit for src/port/ anyway.
This patch is mostly mine, but it owes a good deal to Nathan Bossart
who pressed for a solution to the password length problem and
created a predecessor patch. Also thanks to Peter Eisentraut and
Stephen Frost for ideas and discussion.
Discussion: https://postgr.es/m/
09512C4F-8CB9-4021-B455-
EF4C4F0D55A0@amazon.com
Tom Lane [Thu, 3 Sep 2020 20:52:09 +0000 (16:52 -0400)]
Avoid lockup of a parallel worker when reporting a long error message.
Because sigsetjmp() will restore the initial state with signals blocked,
the code path in bgworker.c for reporting an error and exiting would
execute that way. Usually this is fairly harmless; but if a parallel
worker had an error message exceeding the shared-memory communication
buffer size (16K) it would lock up, because it would wait for a
resume-sending signal from its parallel leader which it would never
detect.
To fix, just unblock signals at the appropriate point.
This can be shown to fail back to 9.6. The lack of parallel query
infrastructure makes it difficult to provide a simple test case for
9.5; but I'm pretty sure the issue exists in some form there as well,
so apply the code change there too.
Vignesh C, reviewed by Bharath Rupireddy, Robert Haas, and myself
Discussion: https://postgr.es/m/CALDaNm1d1hHPZUg3xU4XjtWBOLCrA+-2cJcLpw-cePZ=GgDVfA@mail.gmail.com
Tom Lane [Thu, 3 Sep 2020 16:16:48 +0000 (12:16 -0400)]
Allow records to span multiple lines in pg_hba.conf and pg_ident.conf.
A backslash at the end of a line now causes the next line to be appended
to the current one (effectively, the backslash and newline are discarded).
This allows long HBA entries to be created without legibility problems.
While we're here, get rid of the former hard-wired length limit on
pg_hba.conf lines, by using an expansible StringInfo buffer instead
of a fixed-size local variable.
Since the same code is used to read the ident map file, these changes
apply there as well.
Fabien Coelho, reviewed by Justin Pryzby and David Zhang
Discussion: https://postgr.es/m/alpine.DEB.2.21.
2003251906140.15243@pseudo
Tom Lane [Thu, 3 Sep 2020 15:45:26 +0000 (11:45 -0400)]
Doc: mention packager-supplied tools for server start/stop, initdb, etc.
The majority of our audience is probably using a pre-packaged Postgres
build rather than raw sources. For them, much of runtime.sgml is not
too relevant, and they should be reading the packager's docs instead.
Add some notes pointing that way in appropriate places.
Text by me; thanks to Daniel Gustafsson for review and discussion,
and to Laurenz Albe for an earlier version.
Discussion: https://postgr.es/m/
159430831443.16535.
11360317280100947016@wrigleys.postgresql.org
Peter Eisentraut [Thu, 3 Sep 2020 11:15:53 +0000 (13:15 +0200)]
doc: Make SQL command names in the catalog documentation links
In passing, fix the initdb references to be <application> rather than
<command>, which is what we normally use.
Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Discussion: https://www.postgresql.org/message-id/flat/87mu5xqc11.fsf@wibble.ilmari.org
Peter Eisentraut [Thu, 3 Sep 2020 11:15:53 +0000 (13:15 +0200)]
doc: Add missing cross-links in system catalog documentation
This makes the first mention of a system catalog or view in each
paragraph in the system system catalog and view documentation pages
hyperlinks, for easier navigation.
Also linkify the first mention of pg_hba.conf in pg_hba_file_rules, as
that's more specific and easier to spot than the link to the client
authentication chapter.
Author: Dagfinn Ilmari Mannsåker <ilmari@ilmari.org>
Discussion: https://www.postgresql.org/message-id/flat/87mu5xqc11.fsf@wibble.ilmari.org
Peter Eisentraut [Thu, 3 Sep 2020 10:38:32 +0000 (12:38 +0200)]
Fix XML id to match containing page
This was apparently a typo when this part of the documentation was
first added.
Amit Kapila [Thu, 3 Sep 2020 02:24:07 +0000 (07:54 +0530)]
Add support for streaming to built-in logical replication.
To add support for streaming of in-progress transactions into the
built-in logical replication, we need to do three things:
* Extend the logical replication protocol, so identify in-progress
transactions, and allow adding additional bits of information (e.g.
XID of subtransactions).
* Modify the output plugin (pgoutput) to implement the new stream
API callbacks, by leveraging the extended replication protocol.
* Modify the replication apply worker, to properly handle streamed
in-progress transaction by spilling the data to disk and then
replaying them on commit.
We however must explicitly disable streaming replication during
replication slot creation, even if the plugin supports it. We
don't need to replicate the changes accumulated during this phase,
and moreover we don't have a replication connection open so we
don't have where to send the data anyway.
Author: Tomas Vondra, Dilip Kumar and Amit Kapila
Reviewed-by: Amit Kapila, Kuntal Ghosh and Ajin Cherian
Tested-by: Neha Sharma, Mahendra Singh Thalor and Ajin Cherian
Discussion: https://postgr.es/m/
688b0b7f-2f6c-d827-c27b-
216a8e3ea700@2ndquadrant.com
Tom Lane [Wed, 2 Sep 2020 22:23:56 +0000 (18:23 -0400)]
Add string_to_table() function.
This splits a string at occurrences of a delimiter. It is exactly like
string_to_array() except for producing a set of values instead of an
array of values. Thus, the relationship of these two functions is
the same as between regexp_split_to_table() and regexp_split_to_array().
Although the same results could be had from unnest(string_to_array()),
this is somewhat faster than that, and anyway it seems reasonable to
have it for symmetry with the regexp functions.
Pavel Stehule, reviewed by Peter Smith
Discussion: https://postgr.es/m/CAFj8pRD8HOpjq2TqeTBhSo_QkzjLOhXzGCpKJ4nCs7Y9SQkuPw@mail.gmail.com
Peter Eisentraut [Wed, 2 Sep 2020 13:17:33 +0000 (15:17 +0200)]
Remove unused parameter
unused since
39bd3fd1db6f3aa3764d4a1bebcd71c4e9c00281
Discussion: https://www.postgresql.org/message-id/flat/
511bb100-f829-ba21-2f10-
9f952ec06ead%402ndquadrant.com
Michael Paquier [Wed, 2 Sep 2020 07:59:22 +0000 (16:59 +0900)]
Add access method names to \d[i|m|t]+ in psql
Listing a full set of relations with those psql meta-commands, without a
matching pattern, has never showed the access method associated with
each relation. This commit adds the access method of tables, indexes
and matviews, masking it for relation kinds where it does not apply.
Note that when HIDE_TABLEAM is enabled, the information does not show
up. This is available when connecting to a backend version of at least
12, where table AMs have been introduced.
Author: Georgios Kokolatos
Reviewed-by: Vignesh C, Michael Paquier, Justin Pryzby
Discussion: https://postgr.es/m/svaS1VTOEscES9CLKVTeKItjJP1EEJuBhTsA0ESOdlnbXeQSgycYwVlliL5zt8Jwcfo4ATYDXtEqsExxjkSkkhCSTCL8fnRgaCAJdr0unUg=@protonmail.com
Michael Paquier [Wed, 2 Sep 2020 05:56:59 +0000 (14:56 +0900)]
Fix thinko with definition of REINDEXOPT_MISSING_OK
This had no direct consequences, but let's be consistent and it would be
confusing when adding new flags. Oversight in
1d65416.
Reported-by: Justin Pryzby
Discussion: https://postgr.es/m/
20200902024148.GB20149@telsasoft.com
Fujii Masao [Wed, 2 Sep 2020 01:55:55 +0000 (10:55 +0900)]
Avoid unnecessary acquisition of SyncRepLock in transaction commit time.
In SyncRepWaitForLSN() routine called in transaction commit time,
SyncRepLock is necessary to atomically both check the shared
sync_standbys_defined flag and operate the sync replication wait-queue.
On the other hand, when the flag is false, the lock is not necessary
because the wait-queue is not touched. But due to the changes by
commit
48c9f49265, previously the lock was taken whatever the flag was.
This could cause unnecessary performance overhead in every transaction
commit time. Therefore this commit avoids that unnecessary aquisition
of SyncRepLock.
Author: Fujii Masao
Reviewed-by: Asim Praveen, Masahiko Sawada,
Discussion: https://postgr.es/m/
20200406050332.nsscfqjzk2d57zyx@alap3.anarazel.de
Alvaro Herrera [Wed, 2 Sep 2020 00:43:23 +0000 (20:43 -0400)]
Fix typo in comment
Introduced by
8b08f7d4820f; backpatch to 11.
Discussion: https://postgr.es/m/
20200812214918.GA30353@alvherre.pgsql
Michael Paquier [Wed, 2 Sep 2020 00:08:12 +0000 (09:08 +0900)]
Improve handling of dropped relations for REINDEX DATABASE/SCHEMA/SYSTEM
When multiple relations are reindexed, a scan of pg_class is done first
to build the list of relations to work on. However the REINDEX logic
has never checked if a relation listed still exists when beginning the
work on it, causing for example sudden cache lookup failures.
This commit adds safeguards against dropped relations for REINDEX,
similarly to VACUUM or CLUSTER where we try to open the relation,
ignoring it if it is missing. A new option is added to the REINDEX
routines to control if a missed relation is OK to ignore or not.
An isolation test, based on REINDEX SCHEMA, is added for the concurrent
and non-concurrent cases.
Author: Michael Paquier
Reviewed-by: Anastasia Lubennikova
Discussion: https://postgr.es/m/
20200813043805.GE11663@paquier.xyz
Tom Lane [Tue, 1 Sep 2020 22:40:37 +0000 (18:40 -0400)]
Improve test coverage of ginvacuum.c.
Add a test case that exercises vacuum's deletion of empty GIN
posting pages. Since this is a temp table, it should now work
reliably to delete a bunch of rows and immediately VACUUM.
Before the preceding commit, this would not have had the desired
effect, at least not in parallel regression tests.
Discussion: https://postgr.es/m/
3490536.
1598629609@sss.pgh.pa.us
Tom Lane [Tue, 1 Sep 2020 22:37:12 +0000 (18:37 -0400)]
Set cutoff xmin more aggressively when vacuuming a temporary table.
Since other sessions aren't allowed to look into a temporary table
of our own session, we do not need to worry about the global xmin
horizon when setting the vacuum XID cutoff. Indeed, if we're not
inside a transaction block, we may set oldestXmin to be the next
XID, because there cannot be any in-doubt tuples in a temp table,
nor any tuples that are dead but still visible to some snapshot of
our transaction. (VACUUM, of course, is never inside a transaction
block; but we need to test that because CLUSTER shares the same code.)
This approach allows us to always clean out a temp table completely
during VACUUM, independently of concurrent activity. Aside from
being useful in its own right, that simplifies building reproducible
test cases.
Discussion: https://postgr.es/m/
3490536.
1598629609@sss.pgh.pa.us
Bruce Momjian [Tue, 1 Sep 2020 21:00:10 +0000 (17:00 -0400)]
doc: clarify that max_wal_size is "during" checkpoints
Previous wording was "between".
Reported-by: Pavel Luzanov
Discussion: https://postgr.es/m/
26906a54-d7cb-2f8e-eed7-
e31660024694@postgrespro.ru
Backpatch-through: 9.5
Alvaro Herrera [Tue, 1 Sep 2020 17:40:43 +0000 (13:40 -0400)]
Raise error on concurrent drop of partitioned index
We were already raising an error for DROP INDEX CONCURRENTLY on a
partitioned table, albeit a different and confusing one:
ERROR: DROP INDEX CONCURRENTLY must be first action in transaction
Change that to throw a more comprehensible error:
ERROR: cannot drop partitioned index \"%s\" concurrently
Michael Paquier authored the test case for indexes on temporary
partitioned tables.
Backpatch to 11, where indexes on partitioned tables were added.
Reported-by: Jan Mussler <jan.mussler@zalando.de>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/16594-
d2956ca909585067@postgresql.org
Tom Lane [Tue, 1 Sep 2020 17:14:44 +0000 (13:14 -0400)]
Teach libpq to handle arbitrary-length lines in .pgpass files.
Historically there's been a hard-wired assumption here that no line of
a .pgpass file could be as long as NAMEDATALEN*5 bytes. That's a bit
shaky to start off with, because (a) there's no reason to suppose that
host names fit in NAMEDATALEN, and (b) this figure fails to allow for
backslash escape characters. However, it fails completely if someone
wants to use a very long password, and we're now hearing reports of
people wanting to use "security tokens" that can run up to several
hundred bytes. Another angle is that the file is specified to allow
comment lines, but there's no reason to assume that long comment lines
aren't possible.
Rather than guessing at what might be a more suitable limit, let's
replace the fixed-size buffer with an expansible PQExpBuffer. That
adds one malloc/free cycle to the typical use-case, but that's surely
pretty cheap relative to the I/O this code has to do.
Also, add TAP test cases to exercise this code, because there was no
test coverage before.
This reverts most of commit
2eb3bc588, as there's no longer a need for
a warning message about overlength .pgpass lines. (I kept the explicit
check for comment lines, though.)
In HEAD and v13, this also fixes an oversight in
74a308cf5: there's not
much point in explicit_bzero'ing the line buffer if we only do so in two
of the three exit paths.
Back-patch to all supported branches, except that the test case only
goes back to v10 where src/test/authentication/ was added.
Discussion: https://postgr.es/m/
4187382.
1598909041@sss.pgh.pa.us
Amit Kapila [Tue, 1 Sep 2020 02:41:39 +0000 (08:11 +0530)]
Fix the SharedFileSetUnregister API.
Commit
808e13b282 introduced a few APIs to extend the existing Buffile
interface. In SharedFileSetDeleteOnProcExit, it tries to delete the list
element while traversing the list with 'foreach' construct which makes the
behavior of list traversal unpredictable.
Author: Amit Kapila
Reviewed-by: Dilip Kumar
Tested-by: Dilip Kumar and Neha Sharma
Discussion: https://postgr.es/m/CAA4eK1JhLatVcQ2OvwA_3s0ih6Hx9+kZbq107cXVsSWWukH7vA@mail.gmail.com
Bruce Momjian [Mon, 31 Aug 2020 22:48:38 +0000 (18:48 -0400)]
doc: document how the backup manifest is transferred
Reported-by: Bernd Helmle
Discussion: https://postgr.es/m/
31acf8b0f1f701d53245e0cae38abdf5c3a0d559.camel@oopsware.de
Backpatch-through: 13
Bruce Momjian [Mon, 31 Aug 2020 22:33:37 +0000 (18:33 -0400)]
doc: add commas after 'i.e.' and 'e.g.'
This follows the American format,
https://jakubmarian.com/comma-after-i-e-and-e-g/. There is no intention
of requiring this format for future text, but making existing text
consistent every few years makes sense.
Discussion: https://postgr.es/m/
20200825183619.GA22369@momjian.us
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 21:36:23 +0000 (17:36 -0400)]
pg_upgrade doc: mention saving postgresql.conf.auto files
Also mention files included by postgresql.conf.
Reported-by: Álvaro Herrera
Discussion: https://postgr.es/m/
08AD4526-75AB-457B-B2DD-
099663F28040@yesql.se
Backpatch-through: 9.5
Alvaro Herrera [Mon, 31 Aug 2020 21:04:40 +0000 (17:04 -0400)]
doc: Update partitioning limitation on BEFORE triggers
Reported-by: Erwin Brandstetter <brsaweda@gmail.com>
Discussion: https://postgr.es/m/CAGHENJ6Le7S3qJJx2TvWvTwRNS3N=BtoNeb7AF2rZvfNBMeQcg@mail.gmail.com
Bruce Momjian [Mon, 31 Aug 2020 21:05:53 +0000 (17:05 -0400)]
docs: in mapping SQL to C data types, timestamp isn't a pointer
It is an int64.
Reported-by: ajulien@shaktiware.fr
Discussion: https://postgr.es/m/
159845038271.24995.
15682121015698255155@wrigleys.postgresql.org
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 20:59:59 +0000 (16:59 -0400)]
doc: cross-link file-fdw and CSV config log sections
There is an file-fdw example that reads the server config file, so cross
link them.
Reported-by: Oleg Samoilov
Discussion: https://postgr.es/m/
159800192078.2886.
10431506404995508950@wrigleys.postgresql.org
Backpatch-through: 9.5
Bruce Momjian [Mon, 31 Aug 2020 20:21:03 +0000 (16:21 -0400)]
docs: clarify intermediate certificate creation instructions
Specifically, explain the v3_ca openssl specification.
Discussion: https://postgr.es/m/
20200824175653.GA32411@momjian.us
Backpatch-through: 9.5