Rethink handling of [Prevent|Is]InTransactionBlock in pipeline mode.

Commits f92944137 et al. made IsInTransactionBlock() set the
XACT_FLAGS_NEEDIMMEDIATECOMMIT flag before returning "false",
on the grounds that that kept its API promises equivalent to those of
PreventInTransactionBlock().  This turns out to be a bad idea though,
because it allows an ANALYZE in a pipelined series of commands to
cause an immediate commit, which is unexpected.

Furthermore, if we return "false" then we have another issue,
which is that ANALYZE will decide it's allowed to do internal
commit-and-start-transaction sequences, thus possibly unexpectedly
committing the effects of previous commands in the pipeline.

To fix the latter situation, invent another transaction state flag
XACT_FLAGS_PIPELINING, which explicitly records the fact that we
have executed some extended-protocol command and not yet seen a
commit for it.  Then, require that flag to not be set before allowing
InTransactionBlock() to return "false".

Having done that, we can remove its setting of NEEDIMMEDIATECOMMIT
without fear of causing problems.  This means that the API guarantees
of IsInTransactionBlock now diverge from PreventInTransactionBlock,
which is mildly annoying, but it seems OK given the very limited usage
of IsInTransactionBlock.  (In any case, a caller preferring the old
behavior could always set NEEDIMMEDIATECOMMIT for itself.)

For consistency also require XACT_FLAGS_PIPELINING to not be set
in PreventInTransactionBlock.  This too is meant to prevent commands
such as CREATE DATABASE from silently committing previous commands
in a pipeline.

Per report from Peter Eisentraut.  As before, back-patch to all
supported branches (which sadly no longer includes v10).

Discussion: https://postgr.es/m/65a899dd-aebc-f667-1d0a-abb89ff3abf8@enterprisedb.com

1 files changed, 5 insertions, 4 deletions

diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml
index 20a40dee5fd..b1ebe33791a 100644
--- a/doc/src/sgml/protocol.sgml
+++ b/doc/src/sgml/protocol.sgml
@@ -1093,9 +1093,10 @@ SELCT 1/0;
     implicit <command>ROLLBACK</command> if they failed.  However, there
     are a few DDL commands (such as <command>CREATE DATABASE</command>)
     that cannot be executed inside a transaction block.  If one of
-    these is executed in a pipeline, it will, upon success, force an
-    immediate commit to preserve database consistency.
-    A Sync immediately following one of these has no effect except to
+    these is executed in a pipeline, it will fail unless it is the first
+    command in the pipeline.  Furthermore, upon success it will force an
+    immediate commit to preserve database consistency.  Thus a Sync
+    immediately following one of these commands has no effect except to
     respond with ReadyForQuery.
    </para>
 
@@ -1103,7 +1104,7 @@ SELCT 1/0;
     When using this method, completion of the pipeline must be determined
     by counting ReadyForQuery messages and waiting for that to reach the
     number of Syncs sent.  Counting command completion responses is
-    unreliable, since some of the commands may not be executed and thus not
+    unreliable, since some of the commands may be skipped and thus not
     produce a completion message.
    </para>
   </sect2>