Wednesday, July 27, 2022

Gitlab 500 internal server error while accessing repos

came across this issue when restoring gitlab 8.16.4-ee to a DR site

ERROR : 500 internal server error

while accessing repo
tail -f /var/log/gitlab/gitlab-rails/production.log

OpenSSL::Cipher::CipherError (bad decrypt):
  app/models/project.rb:519:in `import_url'
  app/models/project.rb:555:in `external_import?'
  app/models/project.rb:547:in `import?'
  app/models/project.rb:563:in `import_in_progress?'
  app/controllers/projects_controller.rb:95:in `show'
  lib/gitlab/middleware/multipart.rb:93:in `call'
  lib/gitlab/request_profiler/middleware.rb:15:in `call'
  lib/gitlab/middleware/go.rb:16:in `call'
  lib/gitlab/middleware/readonly_geo.rb:29:in `call'


SOLUTION
It is pretty obvious that it is related to the gitlab secret key

"OpenSSL bad decrypt"

#1. copy /etc/gitlab/gitlab-secrets.json from old instance to new
#2. sudo gitlab-rails runner  "Project.where.not(import_url: nil).each { |p| p.import_data.destroy if p.import_data }"


Gitlab Backups:
## gitlab-rake gitlab:backup:create

You may run into this error -
Dumping PostgreSQL database gitlab_prod ... pg_dump: server version: 9.5.2; pg_dump version: 9.2.18)

just make sure you point pg_dump and psql to latest version

/opt/gitlab/embedded/bin/pg_dump --version
pg_dump (PostgreSQL) 9.6.1

lrwxrwxrwx 1 root root   49 Jan 31 17:10 pg_dump -> /opt/gitlab/embedded/postgresql/9.6.1/bin/pg_dump

lrwxrwxrwx 1 root root       46 Jan 31 17:42 psql -> /opt/gitlab/embedded/postgresql/9.6.1/bin/psql


## make sure you backup /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb seperately