pgpool2.git
14 months agoFixed comments in sample pgpool.conf.
Bo Peng [Mon, 25 Mar 2024 07:10:46 +0000 (16:10 +0900)]
Fixed comments in sample pgpool.conf.

- The comment for sr_check_period. The default value should be 10 seconds.
- Also fixed some typos in comments.

Patch is created by hiroin and modified by Bo Peng.

14 months agoRemove a file under config directory.
Takuma Hoshiai [Thu, 21 Mar 2024 13:16:28 +0000 (22:16 +0900)]
Remove a file under config directory.

Remove Makefile.in etc. generated by autoconf.
Create .gitignore under src/config and add generated files by bison and flex.

14 months agoAllow reset queries to run even if extended queries do not end.
Tatsuo Ishii [Wed, 20 Mar 2024 22:31:29 +0000 (07:31 +0900)]
Allow reset queries to run even if extended queries do not end.

Commit 240c668d "Guard against inappropriate protocol data." caused
reset queries fail if extended query messages do not end. This commit
fix that by checking whether we are running reset queries in
SimpleQuery(). Also add the test case for this.

14 months agoFix a compiler warning
Takuma Hoshiai [Wed, 20 Mar 2024 23:57:23 +0000 (08:57 +0900)]
Fix a compiler warning

Fix warning introduced in the previous commit.

14 months agoFix memory leak pointed out by Coverity.
Takuma Hoshiai [Wed, 20 Mar 2024 16:41:52 +0000 (01:41 +0900)]
Fix memory leak pointed out by Coverity.

14 months agoTest: enhance 082.guard_against_bad_protocol script comment.
Tatsuo Ishii [Mon, 18 Mar 2024 04:50:03 +0000 (13:50 +0900)]
Test: enhance 082.guard_against_bad_protocol script comment.

14 months agoGuard against inappropriate protocol data.
Tatsuo Ishii [Mon, 18 Mar 2024 01:33:16 +0000 (10:33 +0900)]
Guard against inappropriate protocol data.

If a simple query message arrives before a sequence of extended query
messages ends (that is, no sync message arrives or some ready for
query messages corresponding the sync message do not arrive yet),
pgpool could hang.  This is because the query context in the session
context for the simple query is overwritten by the query contexts of
the extended query messages.

This commit implements a guard in SimpleQuery() by checking whether
extended query protocol messages ended. If they do not end, raise a
FATAL error. A known example detected by this checking is JDBC
driver's "autosave=always" option. This means pgpool will not accept
the option after this commit until the issue (sending a simple
protocol message before ending extended query message protocol) is
fixed by the JDBC driver side.

Discussion:
[pgpool-hackers: 4427] Guard against ill mannered frontend
https://www.pgpool.net/pipermail/pgpool-hackers/2024-February/004428.html

14 months agoEnhance the stability of detach_false_primary.
Tatsuo Ishii [Sun, 17 Mar 2024 01:11:04 +0000 (10:11 +0900)]
Enhance the stability of detach_false_primary.

It was possible that enabling detach_false_primary caused that all
backend nodes went down.

Suppose watchdog is enabled and there are 3 watchdog nodes pgpool0,
pgpool1 and pgpool2. If pgpool0 and pgpool1 find primary PostgreSQL
goes down due to network trouble between pgpool and PostgreSQL, they
promote a standby node. pgpool2 could find that there are two primary
nodes because the backend status at pgpool2 has not been synced with
pgpool0 and pgpool1, and pgpool2 perform detach_false_primary against
the standby, which is being promoted.

To prevent the issue, now detach_false_primary is performed only by
watchdog leader node. With this, pgpool will not see half baked
backend status and the issue described above will not happen.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2024-February/004432.html
([pgpool-hackers: 4431] detach_false_primary could make all nodes go down)

14 months agoRevert "Enhance the stability of detach_false_primary."
Tatsuo Ishii [Sun, 17 Mar 2024 00:01:13 +0000 (09:01 +0900)]
Revert "Enhance the stability of detach_false_primary."

This reverts commit c5b25883d21a180ec54a2fea9de67d5da1367464.

This commit accidentally included other updates.

14 months agoEnhance the stability of detach_false_primary.
Tatsuo Ishii [Sat, 16 Mar 2024 13:07:17 +0000 (22:07 +0900)]
Enhance the stability of detach_false_primary.

It was possible that enabling detach_false_primary caused that all
backend node went down.

Suppose watchdog is enabled and there are 3 watchdog nodes pgpool0,
pgpool1 and pgpool2. If pgpool0 and pgpool1 find primary PostgreSQL
goes down due to network trouble between pgpool and PostgreSQL, they
promote a standby node. pgpool2 could find that there are two primary
nodes because the backend status at pgpool2 has not been synced with
pgpool0 and pgpool1, and pgpool2 perform detach_false_primary against
the standby, which is being promoted.

To prevent the situation, now detach_false_primary is performed by only
watchdog leader node. With this, pgpool will not see half baked backend
status and the issue described above will not happen.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2024-February/004432.html
([pgpool-hackers: 4431] detach_false_primary could make all nodes go down)

14 months agoTest: fix test failure introduced in the previous commit.
Bo Peng [Sat, 16 Mar 2024 13:11:23 +0000 (22:11 +0900)]
Test: fix test failure introduced in the previous commit.

14 months agoTest: fixed regression test 005.jdbc to avoid Java 21 deprecation warnings.
Bo Peng [Fri, 15 Mar 2024 04:54:59 +0000 (13:54 +0900)]
Test: fixed regression test 005.jdbc to avoid Java 21 deprecation warnings.

Replace Runtime.exec(String) with Runtime.exec(String[]) to avoid Java 21 deprecation warnings.
Patch is created by Vladimir Petko and modified by Bo Peng.

15 months agoDoc: update copy right.
Bo Peng [Thu, 29 Feb 2024 05:13:13 +0000 (14:13 +0900)]
Doc: update copy right.

15 months agoDoc: add release notes.
Bo Peng [Tue, 27 Feb 2024 02:28:25 +0000 (11:28 +0900)]
Doc: add release notes.

15 months agoRemove restore_command from sample scripts.
Bo Peng [Tue, 27 Feb 2024 00:55:28 +0000 (09:55 +0900)]
Remove restore_command from sample scripts.

Restore_command is not required because replication slot is enabled.
It causes occasional failover failure.

15 months agoFix the default values.
Bo Peng [Mon, 26 Feb 2024 10:43:10 +0000 (19:43 +0900)]
Fix the default values.

Fixed the default values of the following parameters:
- recovery_user
- failover_on_backend_shutdown
- insert_lock

15 months agoDoc: fix Japanese watchdog document.
Tatsuo Ishii [Mon, 26 Feb 2024 07:05:31 +0000 (16:05 +0900)]
Doc: fix Japanese watchdog document.

It mistakenly used "master" watchdog node instead of "leader".

15 months agoDoc: fix English watchdog document.
Tatsuo Ishii [Mon, 26 Feb 2024 07:01:27 +0000 (16:01 +0900)]
Doc: fix English watchdog document.

It mistakenly used "main" PostgreSQL node instead of "primary".

15 months agoFix to use forward declation of a variable.
Tatsuo Ishii [Thu, 22 Feb 2024 11:35:36 +0000 (20:35 +0900)]
Fix to use forward declation of a variable.

It is required by our coding standard (we follow PostgreSQL's coding
standard). Also fix small typo.

15 months agoFix statement_level_load_balance with BEGIN etc.
Tatsuo Ishii [Sat, 10 Feb 2024 02:50:28 +0000 (11:50 +0900)]
Fix statement_level_load_balance with BEGIN etc.

When statement_level_load_balance is enabled,
BEGIN/END/COMMIT/ABORT/SET/SAVEPOINT/RELEASE SAVEPOINT/DEALLOCATE
ALL/DISCARD were sent to primary node and all standby nodes even if
load_balance_mode is off. This is not only plain wrong but caused slow
down if one of the standby nodes are in remote network. Fix this in
that pgpool sends such queries to primary node only when
load_balance_mode is off.

Note that if load_balance_mode is on and statement_level_load_balance
is on, such queries are sent to all nodes as before. This is
necessary. For example, suppose there are 2 PostgreSQL nodes 0 and
1. An explicit transaction starts followed by two read only
SELECTs. The first SELECT is sent to node 0 because the node 0 is
chosen as the load balance node. The second SELECT is sent to node 1
because the node 1 is chosen as the load balance node. If pgpool has
not sent BEGIN to both node 0 and 1 when the transaction started, the
first or the second SELECT will be executed outside the transaction,
which is not an expected behavior. However this may bring slow down
mentioned above. I guess this has been less known to users and I
decided to add some notes to the statement_level_load_balance doc.

Reported: [pgpool-general: 8998] https://www.pgpool.net/pipermail/pgpool-general/2024-January/009059.html
Discussion: [pgpool-hackers: 4422] https://www.pgpool.net/pipermail/pgpool-hackers/2024-February/004423.html
Backpatch-through: v4.1

16 months agoTest: now it will be displayed in the log if segfault occurs.
Bo Peng [Tue, 9 Jan 2024 03:11:48 +0000 (12:11 +0900)]
Test: now it will be displayed in the log if segfault occurs.

17 months agoTest: enhance 037.failover_session/test.sh.
Tatsuo Ishii [Mon, 25 Dec 2023 08:10:12 +0000 (17:10 +0900)]
Test: enhance 037.failover_session/test.sh.

Previously it mistakenly executed pg_ctl stop after starting pgbench
in background.  The smart shutdown always allows pgbench to run
successfully with pgbench -C option not being set because backend
does not shutdown while client session continues. In order to shutdown
backend in the middle of pgbench run, tweak health check parameters so
that it detects the backend down as soon as possible. This will
trigger failover in the middle of pgbench run.

With these changes Pgpool-II 4.5 and beyond succeeds in all 4 tests,
while pre-4.5 will fail in all 4 tests (that was the originally
expected result).

Also allow to run all the 4 tests even if some tests fail so that we
can check which of which test failed.

17 months agoDoc: fix delay_threshold_by_time document mistake.
Bo Peng [Thu, 21 Dec 2023 09:17:51 +0000 (18:17 +0900)]
Doc: fix delay_threshold_by_time document mistake.

Millisecond is correct.

17 months agoStart 4.6 development.
Bo Peng [Tue, 12 Dec 2023 04:30:55 +0000 (13:30 +0900)]
Start 4.6 development.

17 months agoDoc: update 4.5 release note.
Bo Peng [Mon, 11 Dec 2023 07:52:10 +0000 (16:52 +0900)]
Doc: update 4.5 release note.

17 months agoFix 4.5 release note.
Tatsuo Ishii [Fri, 8 Dec 2023 07:27:57 +0000 (16:27 +0900)]
Fix 4.5 release note.

Description of multi-statement was not accurate and could cause misunderstanding.
Also mention that load balance for PREPARE/EXECUTE/DEALLOCATE is now possible.

17 months agoRemove duplicate definition of TransactionId.
Tatsuo Ishii [Thu, 7 Dec 2023 21:03:40 +0000 (06:03 +0900)]
Remove duplicate definition of TransactionId.

Since commit: ca300f839, following is defined in
src/include/parser/pg_list.h and src/include/parser/primnodes.h.

typedef uint32 TransactionId;

This is harmless in moder OS/compilers, but an old RHEL5 user reported
that this results in compile error:
https://www.pgpool.net/pipermail/pgpool-general/2023-December/009040.html

So remove the definition from primnodes.h (remove it from pg_list.h
causes another compile error).

Back patched to V4_5_STABLE.

18 months agoDoc: add release notes of 4.0.25-4.4.5.
Bo Peng [Wed, 29 Nov 2023 06:56:48 +0000 (15:56 +0900)]
Doc: add release notes of 4.0.25-4.4.5.

18 months agoDoc: update Installation document to mention that from Pgpool-II 4.5 it is required...
Bo Peng [Mon, 27 Nov 2023 05:19:18 +0000 (14:19 +0900)]
Doc: update Installation document to mention that from Pgpool-II 4.5 it is required to run "autoreconf -fi" first to generate configure file.

18 months agoDoc: Add Japanese 4.5 release note.
Bo Peng [Mon, 27 Nov 2023 02:49:14 +0000 (11:49 +0900)]
Doc: Add Japanese 4.5 release note.

18 months agoDoc: update Configuration Example "8.2. Pgpool-II + Watchdog Setup Example" to Pgpool...
Bo Peng [Mon, 27 Nov 2023 02:34:57 +0000 (11:34 +0900)]
Doc: update Configuration Example "8.2. Pgpool-II + Watchdog Setup Example" to Pgpool-II 4.5 and PostgreSQL 16.

18 months agoModify the replication slot name conversion in sample scripts to add support for...
Bo Peng [Fri, 17 Nov 2023 03:46:54 +0000 (12:46 +0900)]
Modify the replication slot name conversion in sample scripts to add support for uppercase hostname.

Patch in created by Sheikh Wasiu Al Hasib and modified by Be Peng.

18 months agoUpdate PostgreSQL version to PostgreSQL 16 in sample scripts.
Bo Peng [Mon, 13 Nov 2023 13:26:21 +0000 (22:26 +0900)]
Update PostgreSQL version to PostgreSQL 16 in sample scripts.

18 months agoDoc: update English Configuration Example "8.2. Pgpool-II + Watchdog Setup Example...
Bo Peng [Mon, 13 Nov 2023 00:57:30 +0000 (09:57 +0900)]
Doc: update English Configuration Example "8.2. Pgpool-II + Watchdog Setup Example" to Pgpool-II 4.5 and PostgreSQL 16.

Several enhancements are also added.

18 months agoDoc: update Pgpool-II version and PostgreSQL version in installation section.
Bo Peng [Fri, 10 Nov 2023 08:18:33 +0000 (17:18 +0900)]
Doc: update Pgpool-II version and PostgreSQL version in installation section.

18 months agoEnable AM_MAINTAINER_MODE on master branch.
Bo Peng [Fri, 10 Nov 2023 07:47:19 +0000 (16:47 +0900)]
Enable AM_MAINTAINER_MODE on master branch.

18 months agoDisable AM_MAINTAINER_MODE.
Bo Peng [Fri, 10 Nov 2023 07:45:04 +0000 (16:45 +0900)]
Disable AM_MAINTAINER_MODE.

18 months agoDoc: add 4.5 english release notes.
Bo Peng [Thu, 9 Nov 2023 13:21:59 +0000 (22:21 +0900)]
Doc: add 4.5 english release notes.

18 months agoUpdate Copyright of the previous commit (4bfca73c6788cee498d74e938fa38c38b9abb6a2).
Bo Peng [Thu, 9 Nov 2023 13:15:08 +0000 (22:15 +0900)]
Update Copyright of the previous commit (4bfca73c6788cee498d74e938fa38c38b9abb6a2).

18 months agoDowngrading some normal ERROR messages to DEBUG messages.
Bo Peng [Thu, 9 Nov 2023 12:52:43 +0000 (21:52 +0900)]
Downgrading some normal ERROR messages to DEBUG messages.

The following ERROR messages are downgraded to DEBUG messages.

(1) ERROR:unable to flush data to frontend

(2) ERROR:  unable to read data from frontend
    DETAIL:  EOF encountered with frontend

(3) ERROR:  unable to read data
    DETAIL:  child connection forced to terminate due to client_idle_limit:30 is reached

(1) and (2)
These messages are cuased when the client did not send a terminate message
before disconnecting to pgpool.
For example, when the client process was forcefully terminated, the error occurs.
Although they are harmless, it can sometimes confuse users.

(3)
If we set "client_idle_limit" to a non-zero value, the connection
will be disconnected if it remains idle since the last query.

The disconnection is caused by Pgpool-II settings,
but Pgpool-II handles the log message as an "ERROR".

Because the ERROR messages above are normal messages, I decide to downgrade them.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-June/004351.html

18 months agoUpdate src/redhat/pgpool_socket_dir.patch used by RPM building.
Bo Peng [Thu, 9 Nov 2023 06:02:06 +0000 (15:02 +0900)]
Update src/redhat/pgpool_socket_dir.patch used by RPM building.

19 months agoUpdate parser changes between PostgreSQL 16 BETA1 and 16 RC1.
Chen Ningwei [Mon, 30 Oct 2023 03:00:12 +0000 (12:00 +0900)]
Update parser changes between PostgreSQL 16 BETA1 and 16 RC1.

19 months agoFix breakage in making pgpool_adm.
Tatsuo Ishii [Fri, 6 Oct 2023 21:15:30 +0000 (06:15 +0900)]
Fix breakage in making pgpool_adm.

Since commit f54efc50 "Fix compiler warnings regarding
DEFAULT_PGSOCKET_DIR.", src/sql/pgpool_adm cannot be compiled with
error:
In file included from pgpool_adm.c:35:0:
../../include/pcp/libpcp_ext.h:32:37: fatal error: parser/pg_config_manual.h: No such file or directory
 #include "parser/pg_config_manual.h"

Since pgpool_adm is a PostgreSQL extension module, it should not
include Pgpool-II's pg_config_manual.h.  Rather than fixing pgpool_adm
to allow to include pgpool's pg_config_manual.h, modify libpcp_ext.h
so that it does not include pgpool's pg_config_manual.h if PGPOOL_ADM
is defined.

19 months agoEnhance error messages in escalation.sh.sample.
Bo Peng [Fri, 6 Oct 2023 13:47:18 +0000 (22:47 +0900)]
Enhance error messages in escalation.sh.sample.

19 months agoFix document typo.
Bo Peng [Fri, 6 Oct 2023 13:31:33 +0000 (22:31 +0900)]
Fix document typo.

19 months agoAdd SSH timeout in escalation.sh.sample.
Bo Peng [Fri, 6 Oct 2023 13:30:23 +0000 (22:30 +0900)]
Add SSH timeout in escalation.sh.sample.

19 months agoRemove unnecessary logging line.
Tatsuo Ishii [Wed, 4 Oct 2023 10:24:10 +0000 (19:24 +0900)]
Remove unnecessary logging line.

Debug message was accidentally left.

19 months agoFix compiler warnings regarding DEFAULT_PGSOCKET_DIR.
Tatsuo Ishii [Tue, 3 Oct 2023 01:46:46 +0000 (10:46 +0900)]
Fix compiler warnings regarding DEFAULT_PGSOCKET_DIR.

Commit 3bd6941f accidentally add "#include "pg_config_manual.h", which
should have been "#include parser/pg_config_manual.h".  This caused to
include parser/pg_config_manual.h and pg_config_manual.h, which caused
compiler warning:

../src/include/parser/pg_config_manual.h:227:0: warning: "DEFAULT_PGSOCKET_DIR" redefined [enabled by default]
 #define DEFAULT_PGSOCKET_DIR  "/tmp"

If DEFAULT_PGSOCKET_DIR value was different among these header files.

Reported by: Peng Bo.

20 months agoTest: add test case 080 for commit "Fix Describe() so that it does not abort with...
Tatsuo Ishii [Thu, 28 Sep 2023 22:46:05 +0000 (07:46 +0900)]
Test: add test case 080 for commit "Fix Describe() so that it does not abort with portal created by DECLARE."

20 months agoFix Describe() so that it does not abort with portal created by DECLARE.
Tatsuo Ishii [Thu, 28 Sep 2023 22:29:28 +0000 (07:29 +0900)]
Fix Describe() so that it does not abort with portal created by DECLARE.

When DECLARE foo CURSOR FOR is executed, a portal named "foo" is
automatically created by PostgreSQL. As the portal is not managed by
Pgpool-II, "Describe foo" message failed with "unable to execute
Describe. unable to get the bind message" error.

To fix this, make Describe() creates a dummy write query context so
that the describe message is sent to primary (streaming replication
mode) or all nodes (replication/snapshot isolation mode).

Problem analysis by Heather Lapointe.
Discussion: https://www.pgpool.net/pipermail/pgpool-general/2023-September/008995.html

20 months agoRemove duplicating definition of pg_attribute_format_arg etc.
Tatsuo Ishii [Thu, 21 Sep 2023 01:51:03 +0000 (10:51 +0900)]
Remove duplicating definition of pg_attribute_format_arg etc.

They were defined in both src/include/parser/pg_config_manual.h and
src/include/parser/pool_parser.h".  Remove the definitions from
pg_config_manual.h because pool_parser.h is including such that
definitions originated in PostgreSQL's c.h.

20 months agoRemove duplication definitions of NAMEDATALEN.
Tatsuo Ishii [Wed, 20 Sep 2023 06:07:26 +0000 (15:07 +0900)]
Remove duplication definitions of NAMEDATALEN.

The definition for NAMEDATALEN is now in
src/include/pg_config_manual.h only.  Also replace POOL_NAMEDATALEN
with NAMEDATALEN in src/utils/pool_select_walker.c.

I tried to eliminate copying pool_config_manual.h from PostgreSQL but
it seems it is a little bit difficult to do that at this moment.
There are some definitions like pg_attribute_format_arg were added to
it.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-August/004366.html

20 months agoDoc: fix pcp_node_info docs.
Tatsuo Ishii [Tue, 19 Sep 2023 04:33:22 +0000 (13:33 +0900)]
Doc: fix pcp_node_info docs.

The actual backend status is obtained using PQpingParams, not pg_ping.

20 months agoFix race condition with pgpool main and pcp child process.
Tatsuo Ishii [Mon, 18 Sep 2023 04:15:56 +0000 (13:15 +0900)]
Fix race condition with pgpool main and pcp child process.

When pcp child process is forked off, pgpool main process unblock all
signals.  This could lead to a race condition if a signal is sent to
the pcp process before it sets up signal handlers. Fix is, do not
unblock signals until the signal handlers are set up.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-September/004398.html

20 months agoFix hung up in pcp_exit_handler of pcp child main.
Tatsuo Ishii [Sun, 17 Sep 2023 11:45:46 +0000 (20:45 +0900)]
Fix hung up in pcp_exit_handler of pcp child main.

pcp_exit_handler is responsible for waiting for exiting pcp child
process. I suspect it is hung up in the wait loop in regression test
001.

https://www.pgpool.net/pipermail/pgpool-hackers/2023-September/004397.html

So I changed it so that it uses waitpid(2) with WNOHANG option.

20 months agoTest: attempt to fix 001.load_balance test timeout.
Tatsuo Ishii [Wed, 13 Sep 2023 08:51:21 +0000 (17:51 +0900)]
Test: attempt to fix 001.load_balance test timeout.

Examining the log reveals that the error was caused by bind error:
address already used.  This is strange because SO_REUSEADDR flags is
set to the listening socket.  I guess the cause is, the kernel is not
fast enough to clean up the socket used by previous pgpool process.
To fix this, existing CHECK_TIME_WAIT environment variable parameter
is modified so that it checks the socket by using netstat -tulpn,
rather than netstat -a, which is too slow.

21 months agoDoc: enhance follow primary command manual.
Tatsuo Ishii [Mon, 21 Aug 2023 07:05:40 +0000 (16:05 +0900)]
Doc: enhance follow primary command manual.

Mention that it is necessary to run pcp_attach_node if
pcp_recovery_node is not used.

21 months agoTest: enhance pgpool_setup.
Tatsuo Ishii [Mon, 21 Aug 2023 02:02:09 +0000 (11:02 +0900)]
Test: enhance pgpool_setup.

Enhance failover script generation and follow primary script
generation so that they create better logging.

21 months agoFeature: allow to set delay_threshold_by_time in milliseconds.
Tatsuo Ishii [Sat, 19 Aug 2023 06:44:02 +0000 (15:44 +0900)]
Feature: allow to set delay_threshold_by_time in milliseconds.

Previously it was allowed only in seconds.  Also put some
refactoring. Create new function "check_replication_delay" which
checks the replication delay and returns 0, -1 or -2, depending on "no
delay", "delayed (delay_threshold_by_time)" or "delayed
(delay_threshold)" accordingly. This should simplify the lengthy
if-statement to check the replication delay.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-August/004372.html

21 months agoComment: add comment to child process and auth staffs.
Tatsuo Ishii [Sat, 19 Aug 2023 06:22:51 +0000 (15:22 +0900)]
Comment: add comment to child process and auth staffs.

21 months agoDoc: add release notes.
Chen Ningwei [Tue, 15 Aug 2023 03:31:41 +0000 (12:31 +0900)]
Doc: add release notes.

21 months agoFix covery warning.
Tatsuo Ishii [Wed, 9 Aug 2023 02:10:28 +0000 (11:10 +0900)]
Fix covery warning.

Fix query cache module assigned time_t value to int32 variable. Change
the variable type to int64. Per Coverity.  Also use difftime() to
calculate time_t difference. This is a recommended practice.
https://www.jpcert.or.jp/sc-rules/c-msc05-c.html

21 months agoFix find_primary_node_repeatedly doesn't terminate within search_primary_node_timeout.
Bo Peng [Fri, 4 Aug 2023 04:43:33 +0000 (13:43 +0900)]
Fix find_primary_node_repeatedly doesn't terminate within search_primary_node_timeout.

21 months agoDoc: fix the missing "logger" application name.
Bo Peng [Thu, 3 Aug 2023 07:57:45 +0000 (16:57 +0900)]
Doc: fix the missing "logger" application name.

21 months agoRemove non-standard "//" comments.
Tatsuo Ishii [Tue, 1 Aug 2023 08:20:49 +0000 (17:20 +0900)]
Remove non-standard "//" comments.

21 months agoAdd header include to pgstrcasecmp.c
Chen Ningwei [Tue, 1 Aug 2023 06:36:31 +0000 (15:36 +0900)]
Add header include to pgstrcasecmp.c

21 months agoDoc: clarify the meaning of stop mode.
Tatsuo Ishii [Tue, 1 Aug 2023 02:49:26 +0000 (11:49 +0900)]
Doc: clarify the meaning of stop mode.

22 months agoFeature: Import PostgreSQL 16 BETA1 new parser.
Chen Ningwei [Mon, 31 Jul 2023 05:17:33 +0000 (14:17 +0900)]
Feature: Import PostgreSQL 16 BETA1 new parser.

Major changes of PostgreSQL 16 parser include:

- Add new option DEFAULT to COPY FROM

  COPY ... FROM stdin WITH (default 'xx');

- Allow the STORAGE type to be specified by CREATE TABLE

  CREATE TABLE t1 (
     c1 VARCHAR(10) STORAGE PLAIN,
     c2 TEXT STORAGE EXTENDED
  );

- Add EXPLAIN option GENERIC_PLAN to display the generic plan for a parameterized query

  EXPLAIN (GENERIC_PLAN) SELECT ...;

- Allow subqueries in the FROM clause to omit aliases

  SELECT COUNT(*) FROM (SELECT ... FROM ...);

- Add SQL/JSON constructors

- Add VACUUM options
  SKIP_DATABASE_STATS, ONLY_DATABASE_STATS to skip or update all frozen statistics
  PROCESS_MAIN to only process TOAST tables

  VACUUM (SKIP_DATABASE_STATS);
  VACUUM (PROCESS_MAIN FALSE) t1 ;

22 months agoDoc: remove the incorrect information from release note 4.2.10.
Bo Peng [Wed, 26 Jul 2023 03:55:06 +0000 (12:55 +0900)]
Doc: remove the incorrect information from release note 4.2.10.

Remove "Fix pgpool_recovery extension script. (Tatsuo Ishii)" from release note 4.2.10.

22 months agoAdd new field "load_balance_node" to "SHOW pool_pools" and pcp_proc_info.
Tatsuo Ishii [Sun, 23 Jul 2023 21:04:27 +0000 (06:04 +0900)]
Add new field "load_balance_node" to "SHOW pool_pools" and pcp_proc_info.

The new field is "1" if pgpool process is connected by a client and
the session uses the backend id as a load balance node. Users can
execute the commands to find out if there's any session that uses the
backend as the load balance node. If so, shutting down the backend may
cause session disconnection.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-July/004353.html

Note: I accidentally pushed only doc part of the commit in 4658f84870e8edfd39920f273bab1a12d71d8986.
This is a follow-up commit for actual codes.

22 months agoInstall system exit callback only after initialization of shared memory.
Muhammad Usama [Sun, 23 Jul 2023 17:34:05 +0000 (22:34 +0500)]
Install system exit callback only after initialization of shared memory.

When the on-exit callback gets called because of a failure to acquire
shared memory. The cleanup function can produce a segfault while accessing
process_info, that lives in shared memory.
Although we can also fix this by bailing out from the exit callback when
process_info is NULL but installing the function after successful initialization
of shared memory is a better approach as the rest of the system always assumes
the process_info can never be NULL, and also, there is nothing to
clean up before child processes are spawned.

22 months agoVerify the spare children config values only in dynamic process management mode
Muhammad Usama [Sun, 23 Jul 2023 17:32:22 +0000 (22:32 +0500)]
Verify the spare children config values only in dynamic process management mode

As suggested by Tatsuo Ishii there is no point in verifying the validity of
min and max spare children configurations for static process management mode,
As these configuration values get ignored in static mode anyway.

22 months agoAdd new field "load_balance_node" to "SHOW pool_pools" and pcp_proc_info.
Tatsuo Ishii [Sun, 23 Jul 2023 01:33:49 +0000 (10:33 +0900)]
Add new field "load_balance_node" to "SHOW pool_pools" and pcp_proc_info.

The new field is "1" if pgpool process is connected by a client and
the session uses the backend id as a load balance node. Users can
execute the commands to find out if there's any session that uses the
backend as the load balance node. If so, shutting down the backend may
cause session disconnection.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-July/004353.html

22 months agoDisallowing setting the max_spare_children greater than num_init_children.
Muhammad Usama [Sat, 22 Jul 2023 13:54:31 +0000 (18:54 +0500)]
Disallowing setting the max_spare_children greater than num_init_children.

Config post-processing stage now throws an error if num_init_children is
smaller than max_spare_children.
Commit also adjusts the number of child processes spawned at startup based on
the configured process management strategy.
For the Aggressive strategy, max_spare_children number of processes is spawned;
for the other two strategies, min_spare_children number of children gets
created at startup.

22 months agoMitigate session disconnection issue in failover/failback/backend error.
Tatsuo Ishii [Tue, 18 Jul 2023 01:10:27 +0000 (10:10 +0900)]
Mitigate session disconnection issue in failover/failback/backend error.

Previously Pgpool-II disconnected client sessions in various
cases. This commit tries to avoid some of cases, especially when a
backend goes down and the backend is not either primary (or main node)
nor load balance node.

Suppose we have 3 streaming replication PostgreSQL cluster and the
client uses primary (node 0) and standby 1 (node 1), but does not use
standby 2 (node 2) because the node 2 is not load balance node.  In
this case ideally shutting down node 2 should not disconnect the
session. However the session is disconnected if the session processing
a query while failover.  The reason why session disconnection in
failover is necessary is, there are bunch of places in the source code
something like this:

for (i = 0; i < NUM_BACKENDS; i++)
{
if (!VALID_BACKEND(i))
   continue;
   :
   :

VALID_BACKEND returns true if the backend is not in down status. If
this code is executed while failover, the code may access the backend
socket which is not available any more and will cause troubles
including segfault. So inside VALID_BACKEND, we check whether failover
is performed, and if so, the pgpool child process exits and the
session disconnects. To aovid it, change VALID_BACKEND so that it
waits for completion of failover. For this purpose new function
wait_for_failover_to_finish() is added. It waits for the completion of
failover up to MAX_FAILOVER_WAIT seconds (for it's fixed to 30).  The
change above will prevent unnecessary session disconnection for
existing sessions.

This commit also tries to prevent unnecessary session disconnection
while accepting new sessions. There are multiple places where it could
happen and this commit fixes them:

- accepting new connection from client. In wait_for_new_connections,
  call wait_for_failover_to_finish to wait for completion of
  failover.

- creating new connection to backend. After accepting connection
  request from client and before creating connection to backend, call
  wait_for_failover_to_finish to wait for completion of failover.

- fixing broken socket. pool_get_cp checks whether exiting backend
  connection is broken. If it's broken, sleep 1 second to expect
  failover happens then calls wait_for_failover_to_finish().

- processing an application name. If an application name is included
  in a startup message, pgpool sends query like "SET application_name
  TO foo" to all backend nodes including node 2, which could cause a
  write error. To prevent the error, I modified
  connect_using_existing_connection, which is sending the SET command
  using do_command, so that do_command does not raise an ERROR by
  wrapping it in TRY/CATCH block.

Note that even with all fixes above, I was not able to fix some cases
where pool_write raises error. pool_write is used to write to backend
socket and there are too many places to fix all of them. For now we
need to run "pcp_detach_node 2" before shutdown it. pcp_detach_node
will tell all pgpool child process that node 2 is going down. Even if
a child process does not notice it and writes to backend 2 socket,
there will be no error because node 2 is still alive.

Finally this commit adds new regression test case
037.failover_session.  For the test pgbench is used. There are 2 cases
for continuous session (without -C option) and repeating
connection/disconnection (with -C option) each. So there are 4 causes
in the test:

"=== test1: backend_weight2 = 0 and pgbench without -C option"
"=== test2: backend_weight2 = 0 and pgbench with -C option"
"=== test3: load_balance_mode = off and pgbench without -C option"
"=== test4: load_balance_mode = off and pgbench with -C option"

test2 and test4 requires pcp_detach_node before shutting down node 2.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-July/004352.html

22 months agoFix crash when v2 protocol is used.
Tatsuo Ishii [Mon, 17 Jul 2023 10:27:43 +0000 (19:27 +0900)]
Fix crash when v2 protocol is used.

When read_kind_from_backend() accumulates statistics data, it was not
prepared for v2 protocol case.

Bug report and patch from MCanivez.
https://www.pgpool.net/mantisbt/view.php?id=807
Slightly modified by me.

22 months agoFix PREPARE in multi-statement case.
Tatsuo Ishii [Wed, 12 Jul 2023 07:35:46 +0000 (16:35 +0900)]
Fix PREPARE in multi-statement case.

If multi-statement query includes PREPARE in the second or latter
position, and subsequent bind message uses the prepared statement, it
fails with "unable to bind" error because the prepared statement is not
saved in sent messages.

To fix this if such a case found after parsing the statement, create a
query context for the named statement and add it to the sent message
list.
Discussion: https://www.pgpool.net/pipermail/pgpool-general/2023-July/008931.html

For this new regression test 079..multi_prepare is added.

22 months agoDoc: Enhance SSH public key authentication setting section in "8.2. Pgpool-II + Watch...
Bo Peng [Tue, 11 Jul 2023 06:10:20 +0000 (15:10 +0900)]
Doc: Enhance SSH public key authentication setting section in "8.2. Pgpool-II + Watchdog Setup Example".

22 months agoFix pgproto to work with bind message using params.
Tatsuo Ishii [Sun, 9 Jul 2023 10:09:26 +0000 (19:09 +0900)]
Fix pgproto to work with bind message using params.

Previously pgproto can only process bind messages without params.

22 months agoFix typo in log message.
Bo Peng [Wed, 5 Jul 2023 08:08:04 +0000 (17:08 +0900)]
Fix typo in log message.

23 months agoDoc: fix load balance mode chapter.
Tatsuo Ishii [Thu, 29 Jun 2023 02:27:31 +0000 (11:27 +0900)]
Doc: fix load balance mode chapter.

In some places "streaming replication mode" was written as "native
replication mode".  Also enhance description regarding additional
requirements for load balancing in the streaming replication mode.
Add some indexes.

23 months agoDowngrade log message.
Tatsuo Ishii [Thu, 29 Jun 2023 01:51:14 +0000 (10:51 +0900)]
Downgrade log message.

It is pointed out in https://www.pgpool.net/mantisbt/view.php?id=806
that there are log entries in the log file:

LOG: pool_pending_message_set_flush_request: msg: Parse

The messages were for debugging and the log level should have been a
DEBUG*. So I changed the log level from LOG to DEBUG5.

23 months agoDoc: fix load balance mode chapter.
Tatsuo Ishii [Wed, 28 Jun 2023 02:56:29 +0000 (11:56 +0900)]
Doc: fix load balance mode chapter.

In some places "streaming replication mode" was written as "native
replication mode".  Also enhance description regarding additional
requirements for load balancing in the streaming replication mode.
Add some indexes.

23 months agoDoc: enhance online recovery documentation.
Bo Peng [Tue, 27 Jun 2023 02:46:17 +0000 (11:46 +0900)]
Doc: enhance online recovery documentation.

Mention that 2nd stage of online recovery does not work properly only for multiple pgpool nodes without watchdog enabled.

23 months agoFix missing CTE SEARCH and CYCLE rewrites.
Bo Peng [Fri, 23 Jun 2023 08:05:53 +0000 (17:05 +0900)]
Fix missing CTE SEARCH and CYCLE rewrites.

In native replication mode, Pgpool-II need to rewrite Date/Time functions to timestamp in WRITE queries.
CTE SEARCH and CYCLE rewrites were missing.

23 months agoRemoved duplicate pcp_listen_address setting in src/sample/pgpool.conf.sample-stream.
Bo Peng [Mon, 19 Jun 2023 04:19:00 +0000 (13:19 +0900)]
Removed duplicate pcp_listen_address setting in src/sample/pgpool.conf.sample-stream.

23 months agoPrevent query cache update under shared lock.
Tatsuo Ishii [Wed, 14 Jun 2023 02:17:38 +0000 (11:17 +0900)]
Prevent query cache update under shared lock.

In https://www.pgpool.net/mantisbt/view.php?id=795 it was pointed out
that expired query cache entry can be modified under shared lock. This
could cause shared memory corruption used by query cache. In order to
fix this, we temporarily release the shared lock and then acquire an
exclusive lock before modifying the cache entry. This could create a
window and we need to get the cache entry meta data again to make sure
that the meta data has not been modified by someone else.

Back-patch to V4.4 stable where shared locking for query cache was
introduced.

23 months agoFeature: allow to route queries to a specific backend node for a specific user connec...
Bo Peng [Tue, 13 Jun 2023 05:56:29 +0000 (14:56 +0900)]
Feature: allow to route queries to a specific backend node for a specific user connection.

This commit adds a new parameter "user_redirect_preference_list" to
allow to send SELECT queries to a specific backend node for a
specific user connection.

23 months agoAdd schema qualification to some system catalog inquiry functions.
Tatsuo Ishii [Mon, 12 Jun 2023 00:46:47 +0000 (09:46 +0900)]
Add schema qualification to some system catalog inquiry functions.

There were a few places where schema qualification were not used while
issuing system catalog inquiry.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-June/004346.html

23 months agoFeature: Support mutiple directories specification for pcp_socket_dir.
Chen Ningwei [Tue, 6 Jun 2023 05:38:49 +0000 (14:38 +0900)]
Feature: Support mutiple directories specification for pcp_socket_dir.

23 months agoFix delay_threshold_by_time and prefer_lower_standby_delay.
Tatsuo Ishii [Mon, 5 Jun 2023 11:18:36 +0000 (20:18 +0900)]
Fix delay_threshold_by_time and prefer_lower_standby_delay.

They were broken since delay_threshold_by_time was introduced in 4.4.

- delay_threshold_by_time was not checked in where_to_send. This broke
  load balancing when replication is delayed.

- select_load_balancing_node was broken if both
  delay_threshold_by_time and prefer_lower_standby_delay were enabled.

In order to fix the issue, where_to_send and
select_load_balancing_node are fixed.  Also add test case for
delay_threshold_by_time are added to 033.prefer_lower_standby_delay.

Discussion: https://www.pgpool.net/pipermail/pgpool-general/2023-June/008864.html

2 years agoFix the wrong variable names in replication_mode_recovery_2nd_stage.sample sample...
Bo Peng [Tue, 30 May 2023 13:17:33 +0000 (22:17 +0900)]
Fix the wrong variable names in replication_mode_recovery_2nd_stage.sample sample script.

2 years agoTest: fix occasional 069.memory_leak_extended test failure.
Tatsuo Ishii [Mon, 22 May 2023 07:27:40 +0000 (16:27 +0900)]
Test: fix occasional 069.memory_leak_extended test failure.

It turned out that reason of the occasional test failure is, pgpool
child process is gone before running ps command after pgbench
finishes. The cause is a kind mismatch FATAL error, "DISCARD ALL
cannot be executed within a pipeline". To fix this, run pgbench in
background and get the process size before pgbench finishes.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-May/004338.html

2 years agoAllow to load balance PREPARE/EXECUTE/DEALLOCATE.
Tatsuo Ishii [Fri, 19 May 2023 06:27:57 +0000 (15:27 +0900)]
Allow to load balance PREPARE/EXECUTE/DEALLOCATE.

Previously PREPARE/EXECUTE/DEALLOCATE are not load balanced. In
streaming replication/logical replication mode, they were sent to
always primary node. In native replication/snapshot isolation
mode,they were always sent to all nodes.

Now they can be load balanced if the SQL statement prepared by PREPARE
command is a read only SELECT.

For this purpose following changes were made:

- is_select_query() looks into "query" member if node is
  PrepareStmt. Also second argument "sql" (query string) is not now
  mandatory. If sql is NULL, warning is emitted and this function
  returns false. If allow_sql_comments is off and node is PrepareStmt
  and query is SelectStmt, is_select_query() does not return false
  anymore.

- pool_has_function_call() looks into "query" member if node is
  PrepareStmt.

- Add PREPARE/EXECUTE/DEALLOCATE test cases to 001.load_balance test.

- Add send_prepare() function which is similar to parse_before_bind in
  extended query protocol case to keep up
  disable_load_balance_on_write rule. send_prepare() is called by
  SimpleQuery() when EXECUTE message is sent by frontend in SL mode so
  that it sends PREPARE message to primary node if it has not sent to
  primary because of load balance.  Note that send_prepare() does
  nothing if the clustering mode is other than SL mode. In native
  replication mode or snapshot isolation mode, all backend has the
  same data, and there's no point to keep up
  disable_load_balance_on_write rule.

- Remove descriptions of restrictions regarding load balance for
  PREPARE/EXECUTE/DEALLOCATE in "Restrictions" section in the docs.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-May/004334.html

2 years agoRemove pool_config.c and scan.c from git repository.
Tatsuo Ishii [Fri, 19 May 2023 00:00:29 +0000 (09:00 +0900)]
Remove pool_config.c and scan.c from git repository.

They are generated files and we should not keep in the git repository.

2 years agoDoc: modify release notes.
Bo Peng [Wed, 17 May 2023 12:11:20 +0000 (21:11 +0900)]
Doc: modify release notes.

2 years agoDoc: add release notes.
Bo Peng [Wed, 17 May 2023 11:48:49 +0000 (20:48 +0900)]
Doc: add release notes.

2 years agoTest: stabilize some tests.
Tatsuo Ishii [Fri, 12 May 2023 08:16:01 +0000 (17:16 +0900)]
Test: stabilize some tests.

069.memory_leak_extended, 070.memory_leak_extended_memqcache and
073.pg_terminate_backend uses "sleep 1" to confirm pgpool starting up,
which makes the tests unstable because there's no guarantee that pgpool
becomes ready within 1 second. Use wait_for_pgpool_startup instead to
stabilize the tests.

2 years agoRefactor pool_where_to_send.
Tatsuo Ishii [Mon, 8 May 2023 00:58:17 +0000 (09:58 +0900)]
Refactor pool_where_to_send.

pool_where_to_send() is a module in charge of judging where to a query
in terms of load balancing.  It had been already big and I felt
difficulty to maintain it. Therefore I extracted two relatively large
code path, namely for streaming replication mode and native
replication mode (plus snapshot isolation mode) into
where_to_send_main_replica() and where_to_send_native_replication()
respectively.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-May/004333.html

2 years agoRefactor send_to_where().
Tatsuo Ishii [Sun, 7 May 2023 02:24:45 +0000 (11:24 +0900)]
Refactor send_to_where().

Previously it had lengthy list of recognized query nodes to find out
unrecognized query quickly. The list must be updated if PostgreSQL
adds new query node. Since the list does very small contribution to
performance with large maintenance pain, I decided that keeping the
query node list is not worth the trouble and remove the list.

Also the second argument of send_to_where() is not actually used, I
remove the argument.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2023-May/004332.html