Skip to content

`pg-upgrade` fails to upgrade a standalone Geo tracking DB

mkozono-ansible@mkozono-omnibus4838b-secondary-geo:~$ sudo gitlab-ctl pg-upgrade --target-version=11
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 11.7
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details

Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: crond: 0s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 0s, normally up
Toggling services: OK
There was an error fetching locale and encoding information from the database
Please ensure the database is running and functional before running pg-upgrade
STDOUT:
STDERR: psql: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
== Fatal error ==
Please check error logs
== Reverting ==
Traceback (most recent call last):
    13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
    12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
    11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
    10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
     9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
     8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:189:in `block in load_file'
     7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:244:in `general_upgrade'
     6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:196:in `common_pre_upgrade'
     5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:338:in `get_locale_encoding'
     4: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:78:in `fetch_lc_ctype'
     3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:66:in `run_query'
     2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:14:in `get_command_output'
     1: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:287:in `error!'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/mixlib-shellout-3.0.9/lib/mixlib/shellout.rb:300:in `invalid!': Expected process to exit with [0], but received '2' (Mixlib::ShellOut::ShellCommandFailed)
---- Begin output of gitlab-psql -d postgres -c 'SHOW LC_CTYPE' -q -t ----
STDOUT:
STDERR: psql: could not connect to server: No such file or directory
    Is the server running locally and accepting
    connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
---- End output of gitlab-psql -d postgres -c 'SHOW LC_CTYPE' -q -t ----
Ran gitlab-psql -d postgres -c 'SHOW LC_CTYPE' -q -t returned 2
    12: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
    11: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
    10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
     9: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
     8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
     7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:189:in `block in load_file'
     6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:244:in `general_upgrade'
     5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:196:in `common_pre_upgrade'
     4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:338:in `get_locale_encoding'
     3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:78:in `fetch_lc_ctype'
     2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:66:in `run_query'
     1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:12:in `get_command_output'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in `rescue in get_command_output': GitlabCtl::Errors::ExecutionError (GitlabCtl::Errors::ExecutionError)
    14: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
    13: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
    12: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
    11: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
    10: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
     9: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:189:in `block in load_file'
     8: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:244:in `general_upgrade'
     7: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:196:in `common_pre_upgrade'
     6: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:337:in `get_locale_encoding'
     5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:346:in `rescue in get_locale_encoding'
     4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:525:in `die'
     3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:487:in `revert'
     2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:46:in `tmp_data_dir'
     1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `data_dir'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `realpath': No such file or directory @ realpath_rec - /var/opt/gitlab/postgresql (Errno::ENOENT)

Looking for a fix

  • It appears pg-upgrade assumes the tracking DB has the geo-secondary role, which is not the case: https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/12.9.2+ee.0/files/gitlab-ctl-commands/pg-upgrade.rb#L184
    • Therefore it did not detect a Geo secondary, and attempted to run general_upgrade
    • We don’t want the geo-secondary role on the tracking DB because it does a bunch of things that we explicitly don’t want when the tracking DB stands alone. We just enable the geo-postgresql service.
  • I patched the condition elsif @roles.include?('geo-secondary') => elsif @roles.include?('geo-secondary') || service_enabled?('geo-postgresql') and reran pg-upgrade. I got another error:
<snip>
Initializing the new database: OK
Upgrading the data:Error upgrading the data to version 11.7
STDOUT:
could not write to log file "pg_upgrade_internal.log"
Failure, exiting
STDERR:
Upgrading the data: NOT OK
Traceback (most recent call last):
    10: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
     9: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
     8: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
     7: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
     6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
     5: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:187:in `block in load_file'
     4: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:329:in `geo_secondary_upgrade'
     3: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `run_pg_upgrade'
     2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `raise'
     1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `exception'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl.rb:27:in `initialize': wrong number of arguments (given 1, expected 3) (ArgumentError)
  • And now it appears the PG server is on 11.7 but the data is on 10, so the server can't run.
2020-04-10_08:30:11.87492 FATAL:  database files are incompatible with server
2020-04-10_08:30:11.87495 DETAIL:  The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.
mkozono-ansible@mkozono-omnibus4838b-secondary-geo:~$ sudo gitlab-ctl pg-upgrade --target-version=11
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: OK
The latest version 11.7 is already running, nothing to do
  • And revert-pg-upgrade fails from here:
mkozono-ansible@mkozono-omnibus4838b-secondary-geo:~$ sudo gitlab-ctl revert-pg-upgrade
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: crond: 3111s, normally up
ok: down: grafana: 3110s, normally up
ok: down: logrotate: 3110s, normally up
Toggling services: OK
Checking if we need to downgrade: NOT OK
Traceback (most recent call last):
    8: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
    7: from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
    6: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
    5: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
    4: from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
    3: from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:48:in `block in load_file'
    2: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:46:in `tmp_data_dir'
    1: from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `data_dir'
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:40:in `realpath': No such file or directory @ realpath_rec - /var/opt/gitlab/postgresql (Errno::ENOENT)

Setting ~S2 for now because I don't know of a workaround.

Edited by Michael Kozono