gitlab-ctl reconfigure fails on redis node in HA Geo setup

Summary

During the upgrade of an HA Geo cluster we observed an issue on the redis node that prevented gitlab-ctl reconfigure to complete after running apt-get install gitlab-ee=12.0.9-ee.0. Rerunning gitlab-ctl reconfigure finishes successfully

Steps to reproduce

Follow instructions described in !3577 (merged) until section "**On all other nodes (not the deploy node)**" in the Primary upgrade section (same also occurs on the Secondary)
Upgrade package on redis node (both on primary and secondary)
Run sudo gitlab-ctl reconfigure

What is the current bug behavior?

Reconfigure fails

What is the expected correct behavior?

Reconfigure succeeds.

Relevant logs



Running handlers:
There was an error running gitlab-ctl reconfigure:

execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 400) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
STDOUT:
STDERR: rake aborted!
Redis::CommandError: NOAUTH Authentication required.
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:15:in `block (6 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasksRecipe: gitlab::gitlab-rails
  * execute[clear the gitlab-rails cache] action run
    [execute] rake aborted!
              Redis::CommandError: NOAUTH Authentication required.
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:15:in `block (6 levels) in <top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `loop'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `block (5 levels) in <top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `each'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `block (4 levels) in <top (required)>'
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `block in with'
              /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `with'
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:8:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => cache:clear => cache:clear:redis
              (See full trace by running task with --trace)
    
    ================================================================================
    Error executing action `run` on resource 'execute[clear the gitlab-rails cache]'
    ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '1'
    ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    STDOUT: 
    STDERR: rake aborted!
    Redis::CommandError: NOAUTH Authentication required.
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:15:in `block (6 levels) in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `loop'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:14:in `block (5 levels) in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `each'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:12:in `block (4 levels) in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `block in with'
    /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/redis/wrapper.rb:19:in `with'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/cache.rake:8:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => cache:clear => cache:clear:redis
    (See full trace by running task with --trace)
    ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ----
    Ran /opt/gitlab/bin/gitlab-rake cache:clear returned 1
    
    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb
    
    400: execute "clear the gitlab-rails cache" do
    401:   command "/opt/gitlab/bin/gitlab-rake cache:clear"
    402:   action :nothing
    403:   not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab-rails']['rake_cache_clear'] }
    404: end
    405: 
    
    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:400:in `from_file'
    
    execute("clear the gitlab-rails cache") do
      action [:nothing]
      default_guard_interpreter :execute
      command "/opt/gitlab/bin/gitlab-rake cache:clear"
      backup 5
      declared_type :execute
      cookbook_name "gitlab"
      recipe_name "gitlab-rails"
      user nil
      domain nil
      not_if { #code block }
    end
    
    System Info:
    ------------
    chef_version=14.13.11
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
    program_name=/opt/gitlab/embedded/bin/chef-client
    executable=/opt/gitlab/embedded/bin/chef-client

Details of package version

Provide the package version installation details


PRIMARY HA APP01

external_url 'http://primary-ha.gogitlab.ml'
roles ['application_role']
nginx['enable'] = true
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11
:8075' }
})
gitlab_rails['gitaly_token'] = ''
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server
gitlab_rails['db_password'] = ''
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server
gitlab_rails['redis_password'] = ''

roles ['geo_primary_role']



PRIMARY HA APP02

external_url 'http://primary-ha.gogitlab.ml'
roles ['application_role']
nginx['enable'] = true
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.142.0.11
:8075' }
})
gitlab_rails['gitaly_token'] = ''
gitlab_rails['db_adapter'] = 'postgresql'
gitlab_rails['db_encoding'] = 'unicode'
gitlab_rails['db_host'] = '10.142.0.9' # IP/hostname of database server
gitlab_rails['db_password'] = ''
gitlab_rails['redis_port'] = '6379'
gitlab_rails['redis_host'] = '10.142.0.10' # IP/hostname of Redis server
gitlab_rails['redis_password'] = ''
roles ['geo_primary_role']
gitlab_rails['auto_migrate'] = false


PRIMARY HA GITALY

external_url 'http://gitlab.example.com'
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'http://primary-ha.gogitlab.ml'
gitaly['listen_addr'] = "10.142.0.11:8075"
gitaly['auth_token'] = ''
gitaly['storage'] = [
  { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' },
]


PRIMARY HA PoSTGRESQL

external_url 'http://primary-ha-postgresql.gogitlab.ml'
roles ['geo_primary_role', 'postgres_role']
repmgr['enable'] = false
consul['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
redis_exporter['enable'] = false
gitlab_monitor['enable'] = false
postgresql['listen_address'] = '10.142.0.9'
postgresql['port'] = 5432
postgresql['sql_user_password'] = ''
postgresql['trust_auth_cidr_addresses'] = %w(10.142.0.12/32 10.142.0.13/32)
postgresql['md5_auth_cidr_addresses'] = ['10.142.0.9/32', '10.164.0.30/32']
gitlab_rails['auto_migrate'] = false
postgresql['max_replication_slots'] = 1


PRIMARY HA REDIS

external_url 'http://primary-ha-redis.gogitlab.ml'
redis['enable'] = true
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_monitor['enable'] = false
gitaly['enable'] = false
redis['bind'] = '10.142.0.10'
redis['port'] = 6379
redis['password'] = 'password'
gitlab_rails['auto_migrate'] = false


SECONDARY HA APP01

external_url 'http://secondary-ha.gogitlab.ml'
roles ['geo_secondary_role', 'application_role']
nginx['enable'] = true
gitlab_rails['auto_migrate'] = false
geo_secondary['db_host'] = '10.164.0.31'
geo_secondary['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['db_password'] = ''
gitlab_rails['redis_host'] = '10.164.0.32'
gitlab_rails['redis_password'] = ''
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33
:8075' }
})
gitlab_rails['gitaly_token'] = ''


SECONDARY HA APP02

external_url 'http://secondary-ha.gogitlab.ml'
roles ['geo_secondary_role', 'application_role']
gitlab_rails['auto_migrate'] = false
nginx['enable'] = true
geo_secondary['db_host'] = '10.164.0.31'
geo_secondary['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['db_password'] = ''
gitlab_rails['redis_host'] = '10.164.0.32'
gitlab_rails['redis_password'] = ''
gitaly['enable'] = false
git_data_dirs({
  'default' => { 'path' => '/var/opt/gitlab/git-data', 'gitaly_address' => 'tcp://10.164.0.33
:8075' }
})
gitlab_rails['gitaly_token'] = ''
geo_secondary['auto_migrate'] = false


SECONDARY HA GITALY

external_url 'http://gitlab.example.com'
postgresql['enable'] = false
redis['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
unicorn['enable'] = false
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
gitlab_rails['rake_cache_clear'] = false
gitlab_rails['auto_migrate'] = false
gitlab_rails['internal_api_url'] = 'http://secondary-ha.gogitlab.ml'
gitaly['listen_addr'] = "10.164.0.33:8075"
gitaly['auth_token'] = ''
gitaly['storage'] = [
  { 'name' => 'default', 'path' => '/var/opt/gitlab/git-data' },
]
geo_secondary['auto_migrate'] = false


SECONDARY HA POSTGRESQL

external_url 'http://secondary-ha-postgresql.gogitlab.ml'
roles ['postgres_role']
repmgr['enable'] = false
postgresql['listen_address'] = '10.164.0.30'
postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32', '10.164.0.31/32', '10.164.0.34/32'
, '10.164.0.35/32']
postgresql['hot_standby'] = 'on'
postgresql['sql_user_password'] = ''
gitlab_rails['db_password'] = ''
geo_postgresql['enable'] = false
gitlab_rails['auto_migrate'] = false
geo_secondary['auto_migrate'] = false


SECONDARY HA REDIS

external_url 'http://secondary-ha-redis.gogitlab.ml'
redis['enable'] = true
sidekiq['enable'] = false
gitlab_workhorse['enable'] = false
unicorn['enable'] = false
postgresql['enable'] = false
nginx['enable'] = false
prometheus['enable'] = false
alertmanager['enable'] = false
pgbouncer_exporter['enable'] = false
gitlab_monitor['enable'] = false
gitaly['enable'] = false
redis['bind'] = '10.164.0.32'
redis['port'] = 6379
redis['password'] = ''
gitlab_rails['auto_migrate'] = false
geo_secondary['auto_migrate'] = false


SECONDARY HA TRACKING

external_url 'http://secondary-ha-tracking.gogitlab.ml'
geo_postgresql['enable'] = true
geo_postgresql['listen_address'] = '10.164.0.31'
geo_postgresql['sql_user_password'] = ''
geo_secondary['db_fdw'] = true
geo_postgresql['fdw_external_password'] = ''
geo_postgresql['md5_auth_cidr_addresses'] = ['10.164.0.30/32']
geo_postgresql['trust_auth_cidr_addresses'] = %w(10.164.0.34/32 10.164.0.35/32)
gitlab_rails['db_host'] = '10.164.0.30'
gitlab_rails['auto_migrate'] = false
alertmanager['enable'] = false
consul['enable'] = false
gitaly['enable'] = false
gitlab_monitor['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
node_exporter['enable'] = false
pgbouncer_exporter['enable'] = false
postgresql['enable'] = false
prometheus['enable'] = false
redis['enable'] = false
redis_exporter['enable'] = false
repmgr['enable'] = false
sidekiq['enable'] = false
unicorn['enable'] = false
geo_secondary['auto_migrate'] = false

Edited Sep 17, 2019 by Fabian Zimmer