Gitlab

Setup

We will be running gitlab from the "gitlab-ce" docker image behind our reverse proxy at <your domain>/gitlab.

Copy&Paste the following in /etc/nginx/sites-available/default, inside the main "server" block:

    location /gitlab
    {
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";

        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Accept-Encoding "";
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        proxy_http_version 1.1;

        proxy_connect_timeout     600;
        proxy_send_timeout        600;
        proxy_read_timeout        600;

        client_max_body_size 50m;

        location /gitlab/ {
            rewrite    /gitlab/(.*) /gitlab/$1 break;
            proxy_pass https://127.0.0.1:<ARBITRARY HTTP PORT>/;
        }
    }

Remember to sudo systemctl reload nginx to apply changes.

You can run the container like so:

docker run --detach \
   --env GITLAB_OMNIBUS_CONFIG="external_url 'https://<YOUR DOMAIN>/gitlab'; letsencrypt['enable'] = false" \
   --publish <ARBITRARY HTTP PORT>:80 --publish <ARBITRARY HTTPS PORT>:443 --publish <ARBITRARY SSH PORT>:22 \
   --name gitlab \
   --restart always \
   --volume /<HOST MACHINE PATH>/config:/etc/gitlab \
   --volume /<HOST MACHINE PATH>/logs:/var/log/gitlab \
   --volume /<HOST MACHINE PATH>/data:/var/opt/gitlab \
   --shm-size 256m \
   gitlab/gitlab-ce:17.1.6-ce.0

I didn't care about SSH as HTTPS suited me just fine so I'm not going to cover SSH in detail in this tutorial.

The "external_url" setting allows you to rebase gitlab's URLs to something other than /, which is what you want when you host your gitlab through NGINX' reverse proxy.

After the container starts and the "config" directory is created, create an SSL directory and copy the certificate and key from NGINX:

sudo mkdir -p <HOST MACHINE PATH>/config/ssl
sudo chmod 755 <HOST MACHINE PATH>/config/ssl
sudo cp /etc/letsencrypt/live/<YOUR DOMAIN>/fullchain.pem <HOST MACHINE PATH>/config/ssl/<YOUR DOMAIN>.crt
sudo cp /etc/letsencrypt/live/<YOUR DOMAIN>/privkey.pem <HOST MACHINE PATH>/config/ssl/<YOUR DOMAIN>.key

You should now be able to browse to https://<YOUR DOMAIN>/gitlab and log in as "root" with the initial root password from <HOST MACHINE PATH>/config/initial_root_password.

Remember to set your own root password right away.

Emails

Unfortunately there's no way about setting up that's both free and easy, unless you're happy to do without email notifications and just poll the server.

You have 2 options: set up your own mailserver, or pay for a "mail sender" service like Mailgun or SendGrid. Mail services like GMail basically don't allow themselves to be used as automated mail senders.

Once you have it, just enter the details below in <HOST MACHINE PATH>/config/gitlab.rb and docker exec gitlab gitlab-ctl reconfigure (more details at https://docs.gitlab.com/omnibus/settings/smtp.html):

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "<SMTP SERVER DOMAIN/IP>"
gitlab_rails['smtp_port'] = <SMTP SERVER PORT>
gitlab_rails['smtp_user_name'] = "<SMTP SERVER LOGIN>"
gitlab_rails['smtp_password'] = "<SMTP SERVER PASSWORD>"
gitlab_rails['smtp_domain'] = "<SMTP SERVER DOMAIN>"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true

gitlab_rails['gitlab_email_enabled'] = true

gitlab_rails['gitlab_email_from'] = 'exampleaccount@<SMTP SERVER DOMAIN>'
gitlab_rails['gitlab_email_display_name'] = 'Example Account'
gitlab_rails['gitlab_email_reply_to'] = 'exampleaccount@<SMTP SERVER DOMAIN>'

Project/repository

Troubleshooting notes

Manual DB cleanup

While messing with domains and certificates, gitlab suddently stopped working.

I got Error 500 when browsing to the General setting and some of the project repos, with a bunch of "ActionView::Template::Error" statements followed by tracebacks in gitlab-rails/production.log

It turned out to be the Mailgun (from early tests), Redmine and Jenkins integrations; they pointed to a different domain and the link got broken during the transition.

The only way I managed to fix this was by resetting all the integrations through the database.

Start a postgres shell:

docker exec -it gitlab su -c "/opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production" gitlab-psql

Delete Mailgun:

update application_settings set encrypted_mailgun_signing_key=null, encrypted_mailgun_signing_key_iv=null, mailgun_events_enabled=false;

Delete Jenkins and Redmine:

delete from integrations where type_new='Integrations::Redmine';

delete from integrations where type_new='Integrations::Jenkins';

That did it.

Dumping the whole database was also quite useful:

docker exec gitlab su -c "/opt/gitlab/embedded/bin/pg_dump -h /var/opt/gitlab/postgresql -d gitlabhq_production" gitlab-psql > pgdump.txt

Another trick in the book to speed up debugging is to copy the "config", "data" and "logs" directories and reapply ownerships (chown -R <user>:<group>, dir by dir) over to a VM on my dev PC, restart the container there and point the reverse proxy to it.

Integrity checks

Another thing that happened when moving domains is I was unable to alter any project setting.

I got this in <GITLAB DIRECTORY>/logs/gitlab-rails/production.log

OpenSSL::Cipher::CipherError ():

encryptor (3.0.0) lib/encryptor.rb:98:in `final'
encryptor (3.0.0) lib/encryptor.rb:98:in `crypt'
encryptor (3.0.0) lib/encryptor.rb:49:in `decrypt'
lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
...

It turned out to be a secrets integrity issue:

xxx@xxx:~$ docker exec gitlab gitlab-rake gitlab:doctor:secrets
I, [2024-10-22T14:20:23.854120 #213482]  INFO -- : Checking encrypted values in the database
I, [2024-10-22T14:21:24.044593 #213482]  INFO -- : - Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration failures: 0
I, [2024-10-22T14:21:24.073101 #213482]  INFO -- : - VirtualRegistries::Packages::Maven::Upstream failures: 0
...
I, [2024-10-22T14:21:30.607754 #213482]  INFO -- : - User failures: 0
I, [2024-10-22T14:21:30.814048 #213482]  INFO -- : - ApplicationSetting failures: 1
I, [2024-10-22T14:21:30.833721 #213482]  INFO -- : - SystemHook failures: 0
...
I, [2024-10-22T14:21:36.975221 #213482]  INFO -- : - Ci::Runner failures: 0
I, [2024-10-22T14:21:37.486262 #213482]  INFO -- : - Ci::Build failures: 0
I, [2024-10-22T14:21:37.486434 #213482]  INFO -- : Total: 1 row(s) affected
I, [2024-10-22T14:21:37.486515 #213482]  INFO -- : Done!

This is what fixed it for me:

$ docker exec -it gitlab gitlab-rails dbconsole --database main

psql (14.11)
Type "help" for help.

gitlabhq_production=> UPDATE projects SET runners_token = null, runners_token_encrypted = null;
UPDATE 4
gitlabhq_production=> UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
UPDATE 12
gitlabhq_production=> UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE 1
gitlabhq_production=> UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
UPDATE 1
gitlabhq_production=> UPDATE ci_runners SET token = null, token_encrypted = null;
UPDATE 0

More at: https://docs.gitlab.com/ee/administration/raketasks/check.html#verify-database-values-can-be-decrypted-using-the-current-secrets