How to Install GitLab on a VPS and Connect Your Own GitLab Runner

Before proceeding with the installation, let’s briefly review what we will deploy and why you might need a self-hosted GitLab instance.

GitLab is a collaborative development and DevOps platform built around Git repositories. In addition to hosting source code, it provides user and access management, issues, merge requests, CI/CD, a container registry, and other tools that bring much of the development workflow together in one place.

GitLab is available as a managed cloud service, but it can also be self-hosted, with the entire platform deployed on your own server. In this case, the administrator controls the data, users, backups, updates, and resources on which the system runs.

However, GitLab alone is not enough for CI/CD. GitLab hosts the project and creates pipelines, but a separate component—GitLab Runner—executes the commands defined in .gitlab-ci.yml.

This is the setup we will build today. GitLab will host the repositories, users, and CI/CD configuration, while the Runner will retrieve pipeline jobs and execute them on the server.

The basic workflow is as follows: a developer pushes changes to GitLab, GitLab creates a pipeline, and then a suitable Runner picks up a job and executes the commands defined in .gitlab-ci.yml.

For a small self-hosted deployment, GitLab and a dedicated Runner can be hosted on the same VPS. This is convenient for a learning environment, a small team, or a project with a moderate workload. As the infrastructure grows, the Runner can be moved to a separate server or distributed across multiple nodes.

This article covers GitLab installation, Runner registration, a test pipeline, HTTPS, resource consumption, backup and restore, and troubleshooting stalled CI jobs.

For production use, it is important not only to get GitLab running, but also to check how it behaves under load, where it stores backups, and what happens if the Runner stops picking up jobs.

GitLab and Runner Architecture

Before installation, it is helpful to review the entire architecture. GitLab and Runner are often viewed as a single system, but in practice they have different roles: GitLab manages projects and pipelines, while Runner executes CI jobs.

In this setup, we will start with a single VPS to avoid overcomplicating the infrastructure too early. Later, we will examine when it makes sense to move Runner to a separate server.

Components Involved in Deployment

The main component is GitLab itself.

It stores repositories, users, projects, access settings, CI/CD configuration, and pipeline history.

GitLab Runner runs separately. Its job is to receive a job from GitLab, prepare the execution environment, and run the commands defined in .gitlab-ci.yml.

The pipeline serves as the link between them. It determines which stages must run after a push, merge request, or other event.

For example, a pipeline may consist of three parts:

• Building the application;
• Running tests;
• Deployment.

GitLab manages the execution order and statuses, while the Runner executes the commands.

In addition to these two main components, you will need a domain name, HTTPS, backup storage, and sufficient spare VPS resources.

It is now important to determine exactly where GitLab and the Runner will run.

Where GitLab and GitLab Runner Will Run

For a small deployment, both components can be hosted on the same VPS.

This setup is suitable for a personal GitLab instance, a test environment, or a small team with simple infrastructure where CI jobs do not run too frequently.

In this setup, a single server performs two roles:

• GitLab handles the web interface, Git operations, and internal services;
• Runner executes CI jobs.

This option is easier to configure and less expensive because it does not require a second VPS.

However, it has one limitation: GitLab and Runner share the same CPU, RAM, and disk resources. If a pipeline starts compiling code intensively or running resource-intensive tests, this may affect GitLab itself.

Therefore, we will initially use a single server, then measure actual resource usage and assess how much Runner affects the primary instance.

Why GitLab and the Runner Should Be Logically Separated

Even if they are physically hosted on the same VPS, it is useful to treat them as two separate parts of the system.

GitLab primarily manages data and state: it stores repositories, users, pipelines, and job results.

The Runner, by contrast, executes commands.

This is a fundamental distinction.

If .gitlab-ci.yml specifies that dependencies should be installed, tests run, or an application built, those actions are performed by the Runner.

Because the Runner is directly involved in executing code, its permissions and isolation generally require greater care.

This logical separation provides several benefits:

• It is easier to identify where a problem lies;
• Resources are easier to control;
• Permissions are easier to restrict;
• The Runner can later be moved to a separate VPS without migrating GitLab itself;
• Multiple Runners can be connected for different types of tasks.

Even in a minimal setup, it is best to think of the system this way from the outset: GitLab manages, while the Runner executes.

This makes it easier to understand what happens after a standard git push.

How a Push Leads to CI Job Execution

A developer modifies the code and pushes a commit to GitLab.

GitLab receives the push, updates the repository, and validates the project’s CI/CD configuration.

If the repository contains a .gitlab-ci.yml file and the triggering conditions are met, GitLab creates a pipeline.

Jobs—the specific tasks to be executed—then appear within the pipeline.

For example:

• build;
• test;
• deploy.

GitLab does not run these commands itself. It waits for a suitable Runner.

The Runner connects to GitLab, picks up an available job, executes its commands, and sends the result back.

The resulting workflow is as follows:

Once the job has finished, GitLab receives its status and displays it in the interface as passed, failed, canceled, or another state.

We will later follow this workflow to diagnose stuck CI jobs: first checking the pipeline, then the Runner, and finally the job’s execution environment.

The remaining step is to prepare the information required for installation.

What to Prepare Before Installation

Before starting, it is best to determine the domain, VPS resources, and backup storage location in advance. This will allow you to proceed with the installation without repeatedly stopping to address missing configuration details.

The following table will help:

At this point, the setup is clear: GitLab will be the central hub, while the Runner will act as a separate executor for CI jobs.

Choosing a VPS and Estimating Resource Requirements

Now we need to determine which VPS can reliably run both components.

It is best to allow some headroom from the outset. GitLab is significantly more resource-intensive than a typical website, while Runner adds extra load precisely when a build or test starts. Therefore, we will first estimate GitLab’s requirements, then add Runner’s requirements, and only then finalize the test environment configuration.

Why GitLab Is More Resource-Intensive Than a Typical Web Application

GitLab Self-Managed is not a single web process.

Even when installed on a single server, it runs a whole set of components. In addition to the web interface itself, GitLab uses PostgreSQL for data storage, Redis, Sidekiq background processes, and Gitaly for handling Git repositories.

This means that a single user request may involve several parts of the system.

For example, opening a project requires both the web application and the database, while git clone, fetch, and push operations place additional load on repository storage. Background operations run separately and also consume CPU and RAM.

GitLab is particularly sensitive to disk performance. Gitaly performs intensive repository operations, so the official GitLab documentation recommends SSD-backed storage and advises against using storage with unpredictable performance. 

Resource usage also changes over time. When no one is working with projects, the server may appear almost idle. However, concurrent Git operations, background jobs, and CI workloads can quickly change the situation.

For this reason, a VPS configuration should be selected based not only on the number of users but also on the workload profile.

How Much CPU, RAM, and Disk Space Does a Small GitLab Instance Need?

For a standard single-node installation, GitLab currently specifies 8 vCPUs and 16 GB of RAM as the baseline configuration. The application component requires at least approximately 40 GB of disk space, with additional capacity needed for PostgreSQL and the repositories themselves. 

In other words, a VPS with 2 vCPUs and 2 GB of RAM should not be considered a typical starting point for a production GitLab instance.

For resource-constrained environments, GitLab does support a more compact configuration. The current documentation specifies a minimum of around 8 GB RAMfor a memory-constrained single-node environment, but achieving this requires specifically reducing the resource consumption of individual components. However, GitLab warns that this optimization may adversely affect performance and the behavior of certain features.

We will therefore consider two scenarios:

• 8 vCPU / 16 GB RAM is a reasonable baseline for a standard single-node GitLab deployment;
• approximately 8 GB of RAM — is already a specialized constrained scenario for a personal or very small instance.

Spare disk capacity is especially important. In addition to the installation itself, disk space will be used by repositories, the database, logs, uploads, CI artifacts, packages, and backups.

If you allocate only as much disk space as GitLab uses immediately after installation, problems will arise later as projects and CI workloads grow.

We have estimated the resources required for GitLab itself. Now let us add the second component to the setup: the Runner.

What Resources Does GitLab Runner Consume?

GitLab Runner has no single universal CPU or RAM requirement.

The reason is simple: Runner itself is a relatively lightweight manager, while most of the load is generated by running CI jobs.

For example, a job that checks a few files and prints a line to the console has almost no impact on the server. By contrast, building a large project, installing thousands of dependencies, or running tests can keep several CPUs busy and consume gigabytes of memory for an extended period.

GitLab recommends sizing Runner resources based on:

• The workload of a single CI job;
• The memory usage of the job;
• The number of jobs running concurrently;
• The number of active projects;
• Number of developers running CI concurrently. 

Concurrency is especially important.

If the Runner executes only one job, the server only needs to handle the load of a single build. If four resource-intensive jobs are allowed to run concurrently, they will compete for CPU, RAM, and disk I/O.

Therefore, for our first Runner, we will limit it to one concurrently running Job. This makes it easier to control the load and later compare the state of the VPS before and during the pipeline run.

This also answers the next question: do you need to purchase a separate server for the Runner from the outset?

When GitLab and Runner Can Share a VPS

For a small personal GitLab instance, a training environment, or a small team, a single VPS can serve both roles.

This setup is particularly convenient when pipelines do not run continuously, CI jobs are relatively lightweight, and concurrency is limited.

The advantage is clear: simpler infrastructure. There is no need to maintain a second server solely for a few short builds each day.

However, the trade-off is equally clear.

If Runner consumes nearly all the CPU or memory during a job, GitLab, running on the same server, will have to compete with it for resources. A resource-intensive build may affect the web interface, Git operations, and background processes.

A shared VPS is therefore a good option as long as the workload remains moderate. It makes sense to run Runner separately if:

• Builds regularly place a heavy load on the CPU;
• Jobs require large amounts of RAM;
• Multiple jobs need to run concurrently;
• CI performs intensive disk I/O;
• Runner executes code that should not have access to the GitLab server;
• Runner must be hosted in infrastructure separate from GitLab, such as at another site;
• GitLab itself must remain stable regardless of the state of CI.

In this article, we’ll start with a single VPS. This will not only simplify the installation but also allow us to later empirically measurehow much running Runner changes resource usage.

Configuration Used in This Article

For our test environment, we will use a basic single-node GitLab configuration rather than trying to squeeze it onto the smallest possible server.

We will use:

• 8 vCPU;
• 16 GB RAM;
• 100 GB SSD;
• A supported version of Ubuntu Server;
• One GitLab instance;
• One GitLab Runner;
• No more than one concurrent CI job.

8 vCPUs and 16 GB of RAM align with GitLab’s current baseline recommendation for a single-node installation. The 100 GB SSD includes the capacity buffer we selected; it is not a fixed GitLab requirement, as the actual storage capacity depends on the number and size of repositories, artifacts, packages, and backups. 

This capacity may not be sufficient for resource-intensive CI workloads. In that case, the right approach is not to keep scaling up the entire VPS indefinitely, but to use one or more separate Runners.

Before installation, it is helpful to compare the options in a single table:

All that remains is to check the VPS itself and record its baseline metrics before installation.

Commands for Checking VPS Resources

First, check the number of available CPUs:

   nproc

For more detailed CPU information, run:

   lscpu

Check the memory as follows:

   free -h

This command displays the total amount of RAM, used and available memory, and swap space.

To check swap separately, run:

   swapon --show

Next, check the disks and partitions:

   lsblk

To check the actual available disk space, run:

   df -h

Finally, record the current VPS load:

   uptime

On a clean server, these metrics will serve as a baseline. Later, we will repeat the measurements after installing GitLab and once more while the pipeline is running.

This gives us three states to compare:

For a quick check, we will put the main commands in a separate cheat sheet—an anchor of sorts that makes the information easier to follow.

Checking VPS Resources

Now, let’s prepare the VPS itself: we will install the system dependencies, check the hostname, system time, and network ports, and then proceed with the GitLab installation.

Preparing a VPS for GitLab

What System Dependencies Are Required

The GitLab Linux package installs most of its internal components automatically, so there is no need to manually build PostgreSQL, Redis, or a web server separately for a basic installation.

However, the operating system still requires a few basic packages and services.

The main ones are:

• curl — for downloading the installation script and testing HTTP requests;
• ca-certificates — for proper HTTPS support;
• openssh-server — for SSH access to the VPS and Git operations over SSH;
• tzdata — for system time zone data;
• perl — required by some system tools and dependencies;
• postfix — optional if GitLab needs to send notifications through a local mail transfer agent.

On a provider-hosted VPS, SSH is typically already installed and running. Some of the other packages may also be included in the Ubuntu image, so reinstalling them will simply confirm that they are present.

There is one consideration regarding email: Postfix is not required for a test environment. GitLab can be configured later to use an external SMTP server, which is often more convenient and predictable in production.

Now that the basic package requirements are clear, you can move on to the setting that determines the GitLab URL.

Why GitLab Needs a Proper Hostname and Domain

GitLab is more than just a website. It continually generates project links, clone URLs, callback URLs, and other addresses, so it is best to define the public URL in advance.

Suppose GitLab will be available at:

   gitlab.example.com

The DNS record for this domain must point to the public IP address of our VPS.

The hostname of the Linux server itself could be, for example:

   gitlab-prod-01

These two do not have to match.

The hostname is primarily used by the administrator and the operating system itself, while the public domain is used by users and in the GitLab configuration.

Later, we will set this address as GitLab’s external_url. Therefore, it is best not to install the application using an arbitrary IP address and then unnecessarily reconfigure all the links to use the domain.

Before installation, simply make sure that DNS resolves correctly: gitlab.example.com → VPS IP

The next question is whether the network will allow the required connections.

Which Ports Must Be Accessible

A basic GitLab installation requires several externally accessible ports.

22/TCP is used for SSH. It allows the administrator to connect to the VPS and users to work with Git repositories over SSH.

80/TCP is required for HTTP. Even if only HTTPS is used in production, this port may be needed for initial access, redirects, and domain validation during certificate issuance.

443/TCP is used for HTTPS.

The basic set is as follows:

If SSH on the server has been moved to a nonstandard port, use the actual port number instead.

There is no need to expose internal GitLab services such as PostgreSQL or Redis. In a single-node installation, they run on the server and should not be publicly accessible without a specific reason.

Check both layers: the Ubuntu firewall and the cloud provider’s network firewall or security group. If a port is allowed at only one layer, it may still be inaccessible from outside.

In a more secure deployment, access to administrative interfaces should, of course, be restricted—at a minimum by external IP address and ideally through a VPN. However, security hardening is beyond the scope of this article; we will focus on a basic installation.

That covers the network. Before installation, two less obvious but important considerations remain: time and disk space.

Why It Is Important to Check the Time and Available Disk Space

Accurate system time affects TLS, authentication, logging, CI/CD, and token functionality.

If the VPS clock differs significantly from the actual time, errors may occur that initially appear unrelated to the system clock, such as certificate issues, token expiration problems, or incorrect event timestamps in GitLab.

Ubuntu usually synchronizes the system clock automatically, so you only need to verify that synchronization is working properly.

Available disk space is even more important.

Over time, GitLab accumulates more than just repositories. Disk space is also used by:

• Database;
• Logs;
• Uploads;
• CI artifacts;
• Packages;
• Temporary files;
• Backups.

Some of this data may grow unnoticed. For example, several pipelines may regularly save artifacts that, after a few months, take up more space than the Git repositories themselves.

Before installation, it is therefore useful to record the initial amount of available disk space and then compare it with actual usage.

All the preliminary checks can now be performed on the server.

Server Preparation Commands

First, update the package index and installed packages:

   sudo apt update
sudo apt upgrade -y

Install the basic dependencies:

   sudo apt install -y \
  curl \
  ca-certificates \
  openssh-server \
  tzdata \
  perl

If you plan to send email locally through Postfix:

   sudo apt install -y postfix

For a test environment, you can skip this step and configure an external SMTP server later.

Check SSH:

   systemctl is-active ssh

Now check the hostname:

   hostnamectl

If necessary, set a more descriptive name:

   sudo hostnamectl set-hostname gitlab-prod-01

Check DNS resolution for the public domain:

   getent hosts gitlab.example.com

or:

   dig +short gitlab.example.com

If dig is not installed, you can install it with the dnsutils package.

Check the system time:

   timedatectl

Make sure the system clock is accurate and time synchronization is active.

Now check the available disk space again:

   df -h

Also check the main mount points:

   lsblk

If the server uses UFW, check its status:

   sudo ufw status

A standard setup requires the following rules:

   sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

You can then enable UFW if it is not already in use:

   sudo ufw enable

Before enabling the firewall, it is especially important to make sure SSH access is allowed; otherwise, you could accidentally lock yourself out of the VPS.

Next comes our standard anchor table.

Essential VPS Preparation Commands

After these checks, the server is ready in practice, not just in theory: the network is accessible, the domain is known, the system time is synchronized, and the available disk space has been recorded.

The next step is to install the GitLab Linux package itself, set external_url, and understand exactly what happens during the first gitlab-ctl reconfigure.

Installing GitLab on a VPS

What the GitLab Linux package includes

The GitLab Linux package includes more than just the web interface.

It also installs and configures the components required to run a self-managed GitLab instance. These include:

• The core GitLab application;
• PostgreSQL;
• Redis;
• Sidekiq;
• Gitaly;
• The bundled Nginx server;
• Service processes and management utilities.

Each component has a specific role.

PostgreSQL stores project, user, and configuration data. Redis is used for internal operations and queues. Sidekiq handles background jobs. Gitaly manages Git repositories. Nginx accepts HTTP and HTTPS requests.

This eliminates the need to assemble GitLab manually from multiple separate services.

This is convenient for a small single-node installation: the administrator gets a ready-to-use stack that can be managed using GitLab tools.

The next important point is where its main configuration is stored.

How GitLab Stores Its Own Configuration

The main configuration file for the Linux package is located here:

   /etc/gitlab/gitlab.rb

It defines settings for the entire instance.

You can use gitlab.rb to configure:

• Public URL;
• Built-in NGINX;
• HTTPS;
• PostgreSQL settings;
• Redis;
• SMTP;
• Backups;
• Storage;
• Limits;
• Individual internal services.

Instead of manually editing dozens of configuration files, you define the desired state of GitLab in one place.

For example: external_url “https://gitlab.example.com” or gitlab_rails['gitlab_email_from'] = '[email protected]'

After modifying the file, apply the configuration:

   sudo gitlab-ctl reconfigure

GitLab regenerates the required internal configuration based on gitlab.rb.

For this reason, you should not edit generated configuration files directly. Any such changes may be overwritten the next time you run reconfigure.

The first setting we will need is external_url.

What the external_url Parameter Does

external_url defines GitLab’s public URL.

For example: external_url “https://gitlab.example.com”

GitLab treats this address as the primary URL of the instance.

It is used to generate:

• Links in the web interface;
• HTTP(S) clone URLs;
• Redirects;
• Some callback URLs;
• URLs that GitLab displays to users.

During the initial installation, external_url can be specified directly:

   sudo EXTERNAL_URL="https://gitlab.example.com" apt install gitlab-ce

In this case, the installer automatically writes the required value to /etc/gitlab/gitlab.rb.

If the URL changes later, simply edit the file manually and reapply the configuration.

This brings us to one of GitLab’s main administrative commands: gitlab-ctl reconfigure.

What Happens When You Run gitlab-ctl reconfigure

The command:

   sudo gitlab-ctl reconfigure

brings the active GitLab configuration in line with the state defined in gitlab.rb.

Put simply, we first specify the desired configuration:

   external_url "https://gitlab.example.com"

and then run:

   sudo gitlab-ctl reconfigure

GitLab then updates the relevant settings for its components.

During reconfiguration, GitLab may:

• Create and update configuration files;
• Check directories;
• Change permissions;
• Apply service settings;
• Restart the required components.

Therefore, reconfigure and a regular restart are not the same.

restart simply restarts services that are already configured:

   sudo gitlab-ctl restart

By contrast, reconfigure first updates the configuration and then brings the services in line with it.

Keep this distinction in mind: we will modify gitlab.rb several more times later.

We can now proceed with the installation.

Hands-on: Installing and Running GitLab

In this article, we will use GitLab Community Edition.

First, add the official GitLab repository:

   curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

Next, install the package and set the public URL:

   sudo EXTERNAL_URL="https://gitlab.example.com" apt install gitlab-ce

The installation may take several minutes.

Once the installation is complete, check the main configuration file:

   sudo nano /etc/gitlab/gitlab.rb

For now, we are interested in the following line:

   external_url "https://gitlab.example.com"

If the installer has already added it, no changes are required.

If you edited the configuration manually, apply the changes:

   sudo gitlab-ctl reconfigure

Now check whether the internal services are running:

   sudo gitlab-ctl status

The output should list the main GitLab components with the status run.

You can also view information about the current environment:

   sudo gitlab-rake gitlab:env:info

This information will also be useful later when troubleshooting.

GitLab creates a temporary password for the root user for the initial sign-in.

To view it, run:

   sudo cat /etc/gitlab/initial_root_password

It is best to change this password immediately after signing in for the first time.

Now open the following URL in your browser:

Sign in as the root user.

At this point, GitLab is installed and can be managed using the standard gitlab-ctl commands.

And, as usual, we have prepared another table for you:

GitLab itself is now running, but public access is not yet fully configured.

Configuring a Domain and HTTPS

GitLab is already installed and accessible on the server. The next step is to make public access production-ready: point a domain to it, enable HTTPS, and verify that GitLab works correctly over a secure URL.

There are two main approaches. You can let GitLab manage HTTPS through its built-in Nginx and automatically obtain a certificate, or place a separate reverse proxy in front of it. The first option is simpler and more appropriate for our setup, but we will cover both.

Where the DNS Record Should Point

The public GitLab domain must resolve to the VPS’s IP address.

For example, if you use:

   gitlab.example.com

then you need the following DNS record: gitlab.example.com → public IP address of the VPS

This is typically an A record for IPv4 or an AAAA record for IPv6.

After updating DNS, verify that the domain resolves correctly: dig +short gitlab.example.com or getent hosts gitlab.example.com

If the response returns the IP address of our server, you can proceed.

A DNS record does not enable HTTPS by itself. It only maps the hostname to the server. The next step is to configure GitLab to serve this domain over TLS.

How GitLab Handles HTTPS

In a basic single-node installation, GitLab uses its built-in NGINX server.

It accepts incoming HTTP and HTTPS requests and forwards them to GitLab’s internal components.

The public URL is configured using:

   external_url "https://gitlab.example.com"

When external_url uses https://, GitLab recognizes that the instance must operate over a secure protocol.

There are two options.

The first is for GitLab to obtain and manage the certificate itself.

The second is to terminate HTTPS at an external reverse proxy, with GitLab running behind it.

For a single VPS, the first option is usually simpler.

When to use built-in certificate provisioning and when to use an external reverse proxy

If GitLab runs as a standalone service on a single VPS, it is more convenient to use the built-in mechanism.

In this case, GitLab itself:

• Serves the domain;
• Obtains a TLS certificate;
• Configures the built-in Nginx server;
• Handles certificate renewal.

This reduces the number of separate components and simplifies maintenance.

An external reverse proxy makes sense if the server already hosts multiple applications or if TLS is managed centrally across the infrastructure.

For example:

This approach can be useful if certificates for all services are issued centrally or if GitLab operates within a more complex architecture.

However, this adds another configuration layer: headers, the protocol used by the client, and the real IP address must be forwarded correctly, and GitLab must recognize that the external connection uses HTTPS.

This is unnecessary for our test environment. We will therefore use GitLab’s built-in Nginx server and built-in TLS management.

What Happens During TLS Certificate Renewal

A TLS certificate has a limited validity period, so it cannot simply be issued once and forgotten. We will configure integration with Let’s Encrypt, a certificate authority that issues free, short-lived certificates.

GitLab’s built-in mechanism can automatically check the certificate and renew it before it expires.

For this to work, the domain must continue to point to the VPS, and the server must remain externally accessible on the required ports.

If the DNS settings change, a firewall blocks the port, or the domain no longer points to this server, automatic renewal may fail.

Therefore, after the initial setup, you should verify not only that the certificate is present but also that the renewal mechanism works.

This is especially important in production: GitLab may remain technically operational after the certificate expires, but browsers, integrations, and Git clients will begin treating the connection as insecure.

Now, let’s configure HTTPS.

Hands-on: Connecting a Domain and Verifying HTTPS

First, let’s verify the DNS settings again:

   dig +short gitlab.example.com

The result should match the VPS’s public IP address.

Now open the GitLab configuration file:

   sudo nano /etc/gitlab/gitlab.rb

Check the public URL:

   external_url "https://gitlab.example.com"

To use the built-in certificate provisioning, enable Let’s Encrypt:

   letsencrypt['enable'] = true

If necessary, specify a contact email address:

   letsencrypt['contact_emails'] = ['[email protected]']

The resulting minimal configuration may look like this:

   external_url "https://gitlab.example.com"
letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['[email protected]']

Save the file and apply the settings:

   sudo gitlab-ctl reconfigure

Then check the status of the services:

   sudo gitlab-ctl status

Now inspect the HTTP headers:

   curl -I https://gitlab.example.com

If the certificate was issued correctly, curl should establish a TLS connection without any certificate validation errors.

You can also check the HTTP-to-HTTPS redirect separately:

   curl -I http://gitlab.example.com

With a correct configuration, users should be redirected to the HTTPS version of the site.

Next, open the following URL in a browser:

and verify that the GitLab page loads without any certificate warnings.

Four steps are sufficient to verify the entire process:

GitLab is now accessible at a standard public HTTPS URL.

The next logical step is to create a test project and verify the Git workflow itself rather than the infrastructure: create a repository, make the first commit, and push it to the new GitLab instance.

Creating a Project and Verifying GitLab Functionality

We will not connect a Runner just yet. First, we will make sure that GitLab can create projects, accept commits, and handle pushes properly. Then, if a CI issue arises later, we will already know that the basic Git functionality is working and that we need to look for the cause further down the chain.

How Projects and Repositories Work in GitLab

In GitLab, a project is more than just a Git repository.

A repository stores file and commit history, while a project adds other features around it:

• Members and access permissions;
• Issues;
• Merge requests;
• CI/CD;
• Variables;
• Artifacts;
• Packages;
• Runner settings;
• Pipeline history.

In other words, the repository is one part of the project.

For example, you can create the following project for a test application:

   demo-app

The project will contain a Git repository that a developer can work with using standard commands:

• git clone
• git add
• git commit
• git push

Git itself continues to work as usual. GitLab adds access control, a web interface, and CI/CD on top of it.

This distinction is useful for further troubleshooting. If git push works but the pipeline does not start, the problem is not with the repository itself. In that case, check .gitlab-ci.yml, the Runner, or the CI settings.

Before pushing the first commit, however, you need to choose an authentication method.

Which Authentication Methods to Use for Git

You can work with a GitLab repository over HTTPS or SSH.

An HTTPS URL typically looks like this:

https://gitlab.example.com/user/demo-app.git

Using SSH:

   [email protected]:user/demo-app.git

HTTPS is convenient when it is easier to use GitLab tokens and credentials. However, a standard account password should not be treated as a universal substitute for a token: Git over HTTPS typically uses a personal access token with the appropriate permissions.

SSH works somewhat differently.

A key pair is generated on the developer’s computer:

• the private key remains with the user;
• the public key is added to the user’s GitLab profile.

GitLab can then identify the user when they connect over SSH without requiring them to enter a password each time.

SSH is generally more convenient for regular work with a self-managed GitLab instance, so we will use it from this point forward.

If the developer does not yet have a key, they can generate one on their computer:

   ssh-keygen -t ed25519 -C "[email protected]"

Then display the public key:

   cat ~/.ssh/id_ed25519.pub

It must then be added to the SSH Keys section of the GitLab profile settings.

You can test the connection as follows:

   ssh -T [email protected]

When you connect for the first time, SSH will ask you to confirm the server fingerprint. You should verify it rather than accepting it without checking.

Once authentication is working, you can proceed to the test repository.

Why You Need a Test Repository Before Setting Up CI

It may seem easier to register a Runner and start running pipelines right away. However, if an error occurs, there are too many unknowns to troubleshoot at once.

For example, if a job does not start, the cause could be:

• An error in .gitlab-ci.yml;
• An incompatible Runner;
• Incorrect tags;
• Issues with the executor;
• An error in the script itself;
• Or an issue with the repository or access to it.

It is therefore easier to validate the infrastructure step by step.

For now, our checkpoint is simple:

If this chain works, we have verified the domain, HTTPS, Git repository, user account, and basic authentication.

We can then add CI as the next layer.

Hands-on: Creating a Project and Making Your First Push

Open the GitLab web interface and create a new blank project.

For this example, name it:

   demo-app

For a private work project, you can leave Visibility set to Private.

After the project is created, GitLab displays the repository URL. In this example, we will use SSH:

   [email protected]:user/demo-app.git

Clone the project to your workstation:

   git clone [email protected]:user/demo-app.git

Change to the project directory:

   cd demo-app

If the repository was created completely empty, add a simple README:

   echo "# Demo App" > README.md

Check the status:

   git status

Add the file to the staging area:

   git add README.md

Create the first commit:

   git commit -m "Initial commit"

If Git does not yet have the user’s name and email address, it will prompt you to configure them. To set them globally, run:

   git config --global user.name "Developer"
git config --global user.email "[email protected]"

Then run git commit again.

Now push the changes:

   git push

For a new repository, Git may require you to specify the branch and upstream explicitly. For example:

   git push -u origin main

After the push succeeds, refresh the project page in GitLab. README.md and the newly created commit should now appear in the repository.

This confirms that the basic Git workflow is working.

The following four core commands will be useful for ongoing work:

We now have a working GitLab instance, our own domain, and our first repository accepting changes from a developer. The basic setup is complete.

Understanding How GitLab Runner Works

Now you can set up CI/CD—and this is where GitLab Runner comes in.

At this stage, it is important to understand the role of this service. Otherwise, things can easily become confusing later: GitLab shows the pipeline, the runner appears online, but the job remains pending. To help prevent such situations, let’s first examine the underlying mechanism.

What Is GitLab Runner?

GitLab Runner is a standalone service that executes CI/CD jobs.

GitLab stores the .gitlab-ci.yml file, creates pipelines, and determines which jobs need to run. However, the commands defined in a job are executed by a runner rather than within the main GitLab process.

For example, a pipeline might include the following:

   test:
  script:
    - echo "Run tests"
    - ./run-tests.sh

GitLab creates the test job, and a runner picks it up and executes the commands in the script section.

The easiest way to think of Runner is as an executor: GitLab decides what needs to be done, and Runner carries it out.

A single GitLab instance can have multiple runners. One runner can serve a specific project, another can serve a group of projects, and a third can handle a broader range of tasks. This makes it possible to distribute the CI workload and use different environments for different types of jobs.

How Runner Differs from GitLab

GitLab and Runner work together but perform different tasks.

GitLab is responsible for managing:

• repositories;
• users;
• branches;
• merge requests;
• pipelines;
• CI/CD variables;
• job history and statuses.

Up to €20,000 in EU-Based IaaS Credits

Frankfurt infrastructure, predictable pricing, and expert migration support.

Get

Runner handles execution.

When CI starts, Runner:

• receives the job;
• prepares the working environment;
• retrieves the source code;
• executes commands;
• collects logs;
• uploads artifacts when necessary;
• reports the final status to GitLab.

In other words, GitLab knows that a job needs to be executed, but it does not have to execute the job itself.

This separation is also useful because Runner can be moved and scaled independently. If CI becomes more demanding, you can add a second Runner or move the Runners to separate VPS instances without making changes to the main GitLab instance.

Now, let’s look at how a job is passed from GitLab to Runner.

How a Runner Receives CI Jobs

A Runner does not accept commands indiscriminately from any project.

First, it registers with GitLab and is associated with a specific scope. The Runner then polls GitLab regularly to check for matching jobs.

When a job becomes available, GitLab evaluates several conditions:

• Whether the Runner is available;
• Whether it is authorized to run jobs for the project;
• Whether the tags match;
• Whether it is permitted to run untagged jobs;
• Whether the Runner meets the job requirements.

If all conditions are met, the Runner picks up the job and begins executing it.

In simplified form, the workflow looks like this:

There is another important layer here: the executor. It determines where and how the Runner will execute commands.

What Is an Executor?

A Runner does not define the execution environment itself.

This is the role of an executor—a mechanism that tells the Runner which environment to use for running a job.

Depending on the selected executor, commands can run:

• directly in the operating system shell;
• inside a Docker container;
• in Kubernetes;
• over SSH;
• in other supported environments.

In other words, the Runner is the process that manages execution, while the executor defines how jobs are executed.

For example:

or:

The executor determines the level of isolation, the reproducibility of the environment, and the extent to which CI affects the server itself.

For a small self-hosted GitLab instance, the two options most commonly considered are Shell and Docker.

How the shell executor differs from the Docker executor

The shell executor runs commands directly in the Runner host’s operating system.

If the Runner is installed on the same VPS as GitLab, the job effectively runs on that server.

The advantage of this approach is its simplicity. Docker is not required, the environment is easy to understand, and commands can be run almost as if they were being executed manually over SSH.

However, there is a significant drawback: the job can access the host environment with the permissions of the Runner user. As a result, a faulty or untrusted CI script may affect the server’s file system or installed tools.

The Docker executor works differently.

For each job, a container is created from the specified image. Commands run inside the container, and the environment can be removed when the job is complete.

This provides a cleaner, more reproducible model:

Each job gets its own environment, and project dependencies do not need to be installed directly on the VPS in advance.

Let’s compare the two approaches:

It is important to understand that the Docker executor does not turn the Runner into a fully secure sandbox. For example, enabling privileged mode for a container or mounting sensitive host paths significantly reduces isolation.

However, for a standard CI setup, it still provides much better separation between the job and the VPS itself.

Which executor we’ll use in this article

In this article, we will use the Docker executor.

It is a better fit for our task for several reasons.

First, each job will run in a predictable container environment. We will not need to install all the test project’s dependencies directly on the GitLab VPS.

Second, once a job is complete, there is less risk of leaving stray files or environment changes behind on the server.

Third, this setup is closer to how runners are commonly used in real-world CI.

For now, the runner itself will remain on the same VPS as GitLab. This will keep the infrastructure compact, and later we will be able to measure how many resources CI consumes while running jobs.

The resulting setup will look like this:

That covers the runner fundamentals. The next step is to install GitLab Runner, register it with our GitLab instance, and verify that it appears online and is ready to accept jobs.

Installing Your Own GitLab Runner

Where to Host the Runner

For a small self-hosted GitLab instance, there are two main options.

The first is to install the Runner on the same VPS:

The advantage of this approach is its minimal infrastructure requirements. There is no need for a second server, services are easier to monitor, and a working environment can be set up more quickly.

We already know the drawback: GitLab and CI have to share CPU, RAM, and disk resources.

The second option is to host the Runner separately:

This setup scales better and reduces the impact of builds on the main GitLab instance.

For production environments with resource-intensive pipelines or untrusted code, a separate Runner is generally preferable. For this article, however, we will use a single VPS for now and limit concurrency to one job.

This allows us to start with a compact, functional configuration and then use actual measurements to determine whether a second server is necessary.

Why a Runner Should Not Be Trusted by Default

A Runner executes the contents of a CI job.

This means that any pipeline the Runner is allowed to run effectively provides it with a set of commands to execute.

For example:

   job:
  script:
    - ./build.sh
    - ./run-tests.sh

For GitLab, this is simply a job definition. For the Runner, these are actual commands that must be executed.

It is therefore important to distinguish between two scenarios.

If the Runner serves only your own private project and all changes go through trusted developers, the risk is relatively low.

However, if the same Runner is available to multiple projects or can execute untrusted code, the isolation requirements become significantly more stringent.

In our case, the Docker executor reduces the job’s direct impact on the VPS operating system, but it does not fully resolve the security issue. Configurations that grant the container elevated privileges or access to sensitive host resources are particularly dangerous.

In practice, it is therefore best to treat the Runner as a separate compute environment and grant it only the permissions actually required for CI.

The next question is how GitLab determines that this Runner belongs to our project.

What the Runner Receives from GitLab During Registration

In the current workflow, a Runner is first created in the GitLab interface, after which the server receives the runner authentication token.

This token typically starts with:

   glrt-

It is used to associate a specific Runner with GitLab.

Do not confuse it with the old registration token. The legacy registration token workflow is gradually being phased out, so new installations should use an authentication token from the outset.

The process is as follows:

1. Create a Runner in the GitLab interface;
2. Configure its settings;
3. Obtain the authentication token;
4. Run gitlab-runner register on the VPS;
5. Provide the GitLab URL and token;
6. Select an executor;
7. The local configuration is saved to config.toml.

After registration, the Runner uses this token to authenticate with GitLab and retrieve available jobs.

The token itself must be treated as a secret. If an unauthorized party gains access to the Runner configuration, the token may be exposed, so it must not be published in an article, screenshot, or repository.

A separate job token is used when a specific job runs. This means the CI environment does not need to receive the Runner’s persistent authentication token.

The next step is to understand how GitLab determines which jobs to assign to this Runner.

What Tags Are and Why You Need Them

Tags allow specific jobs to be assigned to specific runners.

For example, a runner can be configured with the following tags:

   docker
linux

Then specify the following in .gitlab-ci.yml:

   test:
  tags:
    - docker
  script:
    - echo "Running tests"

This job should be assigned to a runner that matches the specified tags.

This is especially useful when multiple runners are available.

For example:

Instead of having GitLab send each job to the first available runner, tags let you explicitly define the required environment.

There is also a downside: if a job requires the docker tag but none of the available runners has that tag, the job will remain pending.

This is why tags will later be one of the first things we check when troubleshooting stuck CI jobs.

For our setup, we will use a single tag:

   docker

We will register the runner as a project runner for demo-app.

Hands-on: Installing and Registering a Runner

Because we selected the Docker executor, Docker must first be installed and running on the VPS.

Check:

   docker --version

and:

   sudo systemctl is-active docker

If Docker is not yet installed, install it before registering a runner with the Docker executor.

Now add the official GitLab Runner repository.

Download the installation script:

   curl -L \
  "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" \
  -o script.deb.sh

You can review it before running it:

   less script.deb.sh

Then run it:

   sudo bash script.deb.sh

Install GitLab Runner:

   sudo apt install -y gitlab-runner

Check the version:

   gitlab-runner --version

Now open our project in the GitLab interface.

Go to: Settings → CI/CD → Runners

Create a new project runner.

For example, specify the following:

   Description: demo-docker-runner
Tag: docker

After it is created, GitLab will display an authentication token. Do not insert it directly into the article; use a variable instead:

   export RUNNER_TOKEN="glrt-REPLACE_WITH_REAL_TOKEN"

Now register the runner:

   sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.example.com" \
  --token "$RUNNER_TOKEN" \
  --executor "docker" \
  --docker-image "alpine:latest" \
  --description "demo-docker-runner"

After successful registration, the configuration will appear in:

   /etc/gitlab-runner/config.toml

Check it:

   sudo cat /etc/gitlab-runner/config.toml

The basic configuration will look something like this:

   concurrent = 1
check_interval = 0

[[runners]]
  name = "demo-docker-runner"
  url = "https://gitlab.example.com"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"
    privileged = false
    disable_cache = false

For now, there are several settings of interest.

concurrent = 1 limits the number of jobs that can run simultaneously across the entire runner configuration.

executor = "docker" enables the Docker executor selected earlier.

image = "alpine:latest" sets the default image if a specific job does not specify its own.

privileged = false keeps the container out of privileged mode, which is not required for our test pipeline.

The authentication token is also stored in the local configuration, so the contents of the actual config.toml file must not be published carelessly or included in a screenshot.

Now check the service:

   sudo systemctl status gitlab-runner

For a quick status check:

   sudo systemctl is-active gitlab-runner

The runner can also verify its registration:

   sudo gitlab-runner verify

Then return to GitLab: Settings → CI/CD → Runners

Our demo-docker-runner should appear as an available project runner.

The main commands used at this stage can be summarized in a short table:

Next comes the configuration.

Configuring the runner to execute CI jobs

The runner is already registered and shown as available in GitLab. The next step is to verify its configuration before running the first pipeline.

In the previous section, we covered registration and linked the Runner to the project. Here, the goal is different: to understand which Jobs it can receive, where the commands will be executed, and how many VPS resources CI is allowed to consume. This is especially important in our setup because GitLab and the Runner are currently hosted on the same server.

How GitLab Selects the Right Runner

When a pipeline creates a new job, GitLab does not assign it to a random runner.

It first looks for a runner that meets several criteria. The runner must be active, have access to the relevant project, and meet the job’s requirements.

If tags are specified for the job, GitLab checks them as well.

For example:

   test:
  tags:
    - docker
  script:
    - echo "Running tests"

This job can be picked up by a runner with the docker tag.

If no suitable runner is available, the job does not necessarily fail. It may simply remain pending and wait for the required runner to become available.

This is an important distinction:

• failed usually means that the job started running but encountered an error;
• pending often indicates that a suitable runner has not yet picked up the job.

Therefore, when troubleshooting CI, first check whether a runner has been assigned rather than examining the commands in the script.

Tags are one of the primary mechanisms used in this selection process.

How Tags Work

We already encountered tag assignment when creating the Runner. Now let’s look at how the matching logic works.

Suppose a Runner has the following tags:

   docker
linux

And a job requires:

   tags:
  - docker

This Runner is a match.

If the job requires two tags:

   tags:
  - docker
  - linux

The Runner is still a match because it has both tags.

But if you specify:

   tags:
  - docker
  - gpu

our Runner will no longer be a match because it has not been assigned the gpu tag.

This makes it possible to segment the infrastructure quite precisely.

For example, you could later create several Runners:

• docker — for regular builds;
• deploy — for deployment only;
• gpu — for GPU workloads;
• arm64 — for ARM builds.

A tag does not create isolation or grant permissions by itself. It is simply a mechanism for selecting an appropriate Runner.

For now, a single tag is sufficient for our environment:

   docker

The next point is less obvious but equally important: what happens to the project files after the Runner receives a job.

What Happens to a Job’s Working Directory

After receiving a job, the Runner must prepare the environment in which the commands will be executed.

When using the Docker executor, it starts a container and prepares the project’s working directory inside it. GitLab Runner checks out the repository’s source code into this directory before executing the script.

Within a job, the path is usually located under:

   /builds/

A directory for the specific project is then created under it.

In other words, the pipeline does not work with files in /var/www or any other persistent application directory on the VPS. Instead, it uses its own working copy of the repository.

This is important for two reasons.

First, a CI job should not assume that files have been retained from a previous run unless they were explicitly saved using cache or artifacts.

Second, changes made within the working environment do not automatically become changes to the Git repository itself.

For example:

   npm install
npm test
npm run build

These commands may create hundreds of temporary files, but when the job finishes, those files do not become commits or automatically appear in GitLab.

If the output needs to be passed further down the pipeline, artifacts are typically used. To speed up repeated dependency downloads, use cache.

We will return to this topic in the following sections. For now, the key point is that each job receives a working environment, while the Runner manages its setup and cleanup.

The next step is to limit this environment’s resource consumption.

What Resource Limits Should Be Set for the Runner

Because GitLab and the Runner are hosted on the same VPS, allowing CI to use all available resources would be a bad idea.

Suppose the server has:

1. 8 vCPUs
2. 16 GB of RAM

If a single job can consume all 8 vCPUs and nearly all the available memory, GitLab will also be left with no resources in reserve.

Therefore, it makes sense to start with conservative limits for our test Runner.

For example:

1. CPU: up to 2 vCPUs
2. RAM: up to 4 GB
3. Concurrent jobs: 1

These are not universal production values. They are intended specifically for our test environment, so that CI cannot accidentally deprive GitLab of resources on the VPS they share.

There are two levels of limits.

The concurrent setting specifies the total number of jobs that a GitLab Runner instance can execute simultaneously.

For example:

   concurrent = 1

If there are multiple Runners, you can also use limit within a specific [[runners]] section.

For the Docker executor, you can also limit the container’s resources:

   cpus = "2"
memory = "4g"

As a result, a single CI job cannot monopolize all the server’s resources.

If it later turns out that the build really needs more resources, the limits can be increased—or the Runner can be moved to a separate VPS.

Now, let’s combine these settings into a working configuration.

Hands-on: Reviewing and Configuring config.toml

The main Runner configuration file is located here:

   /etc/gitlab-runner/config.toml

Before making changes, it is a good idea to create a backup:

   sudo cp /etc/gitlab-runner/config.toml \
  /etc/gitlab-runner/config.toml.bak

Now open the file:

   sudo nano /etc/gitlab-runner/config.toml

After registration, the file should already contain a section for our Runner.

For our test environment, configure it approximately as follows:

   concurrent = 1
check_interval = 0

[[runners]]
  name = "demo-docker-runner"
  url = "https://gitlab.example.com"
  executor = "docker"
  limit = 1
  [runners.docker]
    image = "alpine:latest"
    privileged = false
    cpus = "2"
    memory = "4g"
    disable_cache = false
    volumes = ["/cache"]

The actual file will also contain an authentication token, but you should not copy it into documentation, a repository, or screenshots.

We will cover only the parameters that are relevant to our test environment.

concurrent = 1 prevents this Runner instance from running multiple jobs simultaneously.

   limit = 1

additionally limits this specific Runner to one job.

The following parameter:

   executor = "docker"

retains the Docker executor selected earlier.

Default image:

   image = "alpine:latest"

will be used if the job does not specify a different image in .gitlab-ci.yml.

For a production project, it is better to pin a specific image tag, such as a particular Alpine version, rather than continuously tracking latest. This makes the CI environment more reproducible.

Next are the resource limits:

   cpus = "2"
memory = "4g"

These prevent a single job container from consuming all the VPS resources.

Leave privileged = false unchanged:

   privileged = false

Privileged mode gives the container significantly broader permissions and should not be enabled unless necessary.

After saving the file, verify the configuration:

   sudo gitlab-runner verify

Then restart the service:

   sudo systemctl restart gitlab-runner

Make sure it is active again:

   sudo systemctl is-active gitlab-runner

To view the latest service messages:

   sudo journalctl -u gitlab-runner -n 50 --no-pager

For quick reference, the key parameters can be summarized in a table:

The Runner is now not only registered but also ready for actual use: it receives only suitable jobs, runs them through Docker, and prevents a single build from consuming all the VPS resources.

The next step is to test the entire workflow. We will create our first .gitlab-ci.yml file, push it to the repository, and see how GitLab creates a pipeline, the Runner picks up the job, and the Docker executor runs the commands inside a container.

Creating Your First GitLab CI Pipeline

What Is .gitlab-ci.yml?

The main GitLab CI/CD configuration is stored in the following file:

   .gitlab-ci.yml

It is usually located in the repository root.

This file defines what happens during a pipeline: which jobs to run, in what order, which container image to use, and which commands to execute.

For example:

   test:
  script:
    - echo "Hello from GitLab CI"

This defines a single job named test.

When GitLab detects this file and the conditions for starting the pipeline are met, it creates a pipeline and assigns the job to a suitable runner.

Therefore, .gitlab-ci.yml can be thought of as a CI/CD script that is stored with the project code and modified through regular commits.

Another advantage is that the pipeline configuration history is preserved in Git. If someone changes the build process, you can see when the change was made and in which commit.

However, even a simple .gitlab-ci.yml file has several levels: the pipeline, stages, and individual jobs. It is best to distinguish between them from the outset.

How pipelines, stages, and jobs are related

A pipeline is the entire CI/CD run for a specific event.

For example, after a git push, GitLab can create a new pipeline.

A pipeline contains stages, which are logical phases of execution.

A typical sequence looks like this:

Each stage, in turn, contains jobs.

For example:

   stages:
  - build
  - test
build_app:
  stage: build
  script:
    - echo "Build"
run_tests:
  stage: test
  script:
    - echo "Tests"

Here:

• the pipeline is the entire run;
• build and test are stages;
• build_app and run_tests are jobs.

By default, the next stage starts after the previous one completes successfully. Multiple jobs within the same stage can run in parallel if runners are available and the required concurrency is allowed.

For the initial test, we do not need a complex workflow yet. One stage and one job are enough, making it easier to verify the connection between GitLab and the runner.

How the Runner Receives Commands from the Pipeline

When GitLab reads .gitlab-ci.yml, it does not send the entire file to the Runner and tell it to figure everything out.

GitLab first processes the CI configuration, creates a pipeline, and generates specific jobs.

The Runner then receives a job that matches its configuration.

In our case, the job will include the following tag:

   tags:
  - docker

The self-hosted Runner was previously assigned the same tag.

When the Runner picks up the job, the Docker executor creates a container from the specified image and runs the commands defined in script.

If you specify:

   image: alpine:3.22
test:
  tags:
    - docker
  script:
    - echo "Runner is working"
    - uname -a

The Runner will execute both commands sequentially inside the Alpine container.

Their output will be sent back to the GitLab job log.

This is why the pipeline log is one of the primary diagnostic tools: it shows not only the final status but also the step at which execution stopped.

Now let’s see what happens after a regular push.

What Happens to the Pipeline After a Push

Suppose we added .gitlab-ci.yml, made a commit, and ran:

   git push

GitLab receives the new commit and validates the CI configuration.

If the file is valid, a pipeline is created.

Several states are then possible.

Initially, the job may be pending: GitLab has created the job, but a Runner has not started executing it yet.

When a Runner picks up the job, its status changes to running.

Once execution is complete, we may see, for example:

• passed — the job completed successfully;
• failed — one of the commands exited with an error;
• canceled — execution was canceled;
• skipped — the job was skipped based on the configuration rules.

The fact that a pipeline appears means that GitLab has read the .gitlab-ci.yml file. Successful job execution also confirms that a Runner was found, the executor is working, and the container was able to execute the commands.

That is exactly what we will verify now.

Hands-on: Creating a Test .gitlab-ci.yml

Navigate to the local demo-app directory:

   cd demo-app

Create the file:

   nano .gitlab-ci.yml

For the first pipeline, use the following configuration:

   stages:
  - test
runner_check:
  stage: test
  image: alpine:3.22
  tags:
    - docker
  script:
    - echo "GitLab Runner is working"
    - 'echo "Project: $CI_PROJECT_PATH"'
    - 'echo "Commit: $CI_COMMIT_SHORT_SHA"'
    - uname -a

Several things happen here.

stages declares a single stage:

   stages:
  - test

runner_check is the job name.

The following parameter:

   stage: test

assigns the job to our stage.

Next, select the container image:

   image: alpine:3.22

We intentionally pin a specific version instead of using latest so that the test environment does not change unexpectedly when the image is updated.

The tag:

   tags:
  - docker

tells GitLab that this job requires a runner with the corresponding tag.

Finally, script contains the commands to run.

The variables:

   CI_PROJECT_PATH
CI_COMMIT_SHORT_SHA

are automatically passed to the job by GitLab. As a result, the log will immediately show which project and commit the pipeline ran for.

Before pushing, it is useful to validate the CI configuration using GitLab’s built-in CI/CD editor or CI Lint. This helps detect YAML errors and some configuration errors before the runner starts.

If validation succeeds, add the file to Git:

   git add .gitlab-ci.yml

Create a commit:

   git commit -m "Add first GitLab CI pipeline"

Then push the changes:

   git push

After pushing, open the project in GitLab and go to Build → Pipelines.

A new pipeline should appear.

Open it and navigate to the runner_check job.

The log should contain output similar to the following:

   GitLab Runner is working
Project: user/demo-app
Commit: a1b2c3d

followed by the output of uname -a.

If the job completes with a passed status, the core CI workflow is working:

The main sequence can be presented in a small table:

The first complete CI workflow is now ready. However, a single job does not closely resemble a real development workflow: an application is typically built first, then tested, and only then deployed.

In the next section, we will make the pipeline slightly more complex by adding several stages and examining how GitLab manages the build → test → deploy sequence without turning .gitlab-ci.yml into a cumbersome configuration.

Adding Multiple Stages to the Pipeline

Why Separate Build, Test, and Deploy

Technically, a pipeline can still work if all commands are placed in a single job.

For example:

However, this approach quickly becomes cumbersome.

First, it becomes harder to determine exactly where an error occurred. If the job fails, you have to find the relevant section in a long log.

Second, different stages often require different environments. The build may require one image, testing another, while deployment may need to run on a dedicated runner.

For this reason, a pipeline is typically divided into logical stages: build → test → deploy

Each stage handles a specific part of the process.

The build stage prepares the application or artifacts.

The test stage verifies the result.

The deploy stage runs only after the preceding checks have completed successfully.

This type of pipeline is easier to read and maintain, especially as the number of jobs grows.

In what order are stages executed?

The stage order is defined explicitly:

GitLab processes this list from top to bottom.

The jobs in the build stage run first.

Once they complete successfully, the test stage begins.

Only then does the deploy stage begin.

If a stage contains multiple jobs, the next stage usually waits for all required jobs in the current stage to complete.

For example:

This allows work to be divided within a stage while preserving the overall pipeline order.

The next question is whether these jobs will run sequentially or concurrently.

When jobs within the same stage run in parallel

Jobs within the same stage can run in parallel if Runners are available and the configured concurrency allows it.

For example:

   unit_tests:
  stage: test
  script:
    - echo "Unit tests"
lint:
  stage: test
  script:
    - echo "Lint"

Both jobs belong to the test stage.

If two Runners are available, or if one Runner can execute two jobs simultaneously, GitLab can run them in parallel.

However, our configuration was previously set to concurrent = 1 and limit = 1.

Therefore, even two jobs in the same stage will run sequentially.

This is intentional: GitLab and the Runner are hosted on the same VPS, and for now, we do not want two containers competing with GitLab for CPU and RAM at the same time.

Later, if the Runner is moved to a separate server, the concurrency can be increased.

What Happens If a Job Fails

By default, a failure in a required job prevents the pipeline from proceeding.

Suppose the pipeline is structured as follows:

If the build job finishes with a failed status, the test stage will usually not run.

If an error occurs during the test stage, the pipeline will not proceed to deploy.

This is one of the main reasons for dividing the process into stages: deployment should not run if the application has not been built or the tests have failed.

You can change this behavior if necessary, for example by using allow_failure, but this option should be used cautiously for standard checks.

For example:

   lint:
  stage: test
  allow_failure: true
  script:
    - ./lint.sh

This job can fail, but the pipeline will continue running.

This is generally not appropriate for critical tests. If a test shows that the application does not work at all, the failure should be treated as blocking.

Now, let’s create a simple pipeline with three stages and examine its graph.

Hands-on: Creating a Multi-Stage Pipeline

Open .gitlab-ci.yml:

   nano .gitlab-ci.yml

Replace the previous test configuration with a more complete one:

   stages:
  - build
  - test
  - deploy
build_demo:
  stage: build
  image: alpine:3.22
  tags:
    - docker
  script:
    - echo "Building demo application"
    - mkdir -p build
    - echo "Demo artifact" > build/app.txt
  artifacts:
    paths:
      - build/
    expire_in: 1 hour
test_demo:
  stage: test
  image: alpine:3.22
  tags:
    - docker
  script:
    - echo "Checking build artifact"
    - test -f build/app.txt
    - cat build/app.txt
deploy_demo:
  stage: deploy
  image: alpine:3.22
  tags:
    - docker
  script:
    - echo "Demo deploy completed"

This introduces the first dependency between stages.

The build_demo job creates the following file:

   build/app.txt

and saves the build/ directory as an artifact.

The artifact is required because the next job runs in a new environment. It cannot assume that the working files from the previous container will somehow remain available.

In test_demo, verify that the artifact is available:

   test -f build/app.txt

If the file does not exist, the command will fail, and the job will be marked as failed.

In that case, the pipeline will not proceed to deploy_demo.

The final stage does not actually deploy anything yet:

   echo "Demo deploy completed"

This is intentional. For now, the goal is to verify the build → test → deploy sequence without giving the test pipeline access to the production server.

Save the file and review the changes:

   git diff

Stage the configuration:

   git add .gitlab-ci.yml

Create a commit:

   git commit -m "Add multi-stage CI pipeline"

Then push it:

   git push

After pushing, go to Build → Pipelines in GitLab.

Open the new pipeline.

The graph should show three sequential stages: build → test → deploy

Within them, you should see: build_demo → test_demo → deploy_demo

The pipeline now resembles a real CI process: one job produces an output, the next verifies it, and deployment runs only after the preceding stages complete successfully.

Working with CI/CD Variables and Secrets

The multi-stage pipeline is already working: GitLab receives a commit, creates stages, the Runner executes a job, and artifacts are passed between stages. Now you can add the next important component— variables and secrets.

Real-world CI pipelines almost always require values that should not be stored carelessly alongside the code, such as tokens, passwords, deployment environment addresses, API keys, and other sensitive parameters. GitLab lets you pass these values to a job separately from .gitlab-ci.yml.

Why Secrets Should Not Be Stored in .gitlab-ci.yml

The .gitlab-ci.yml file is stored in the Git repository.

This means that everything written in it:

• Is added to the commit history;
• May be visible to other project members;
• Remains there even after the line is subsequently deleted;
• May accidentally be included in a fork, backup, or export.

For example, the following is a bad approach:

   deploy:
  script:
    - export API_TOKEN="super-secret-token"
    - ./deploy.sh

Even if the token is later removed from the file, it will remain in the Git history.

The same applies to:

   variables:
  DB_PASSWORD: "secret-password"

or:

   script:
  - 'curl -H "Authorization: Bearer abc123..."'

Secrets should be stored separately from the repository and passed to the job only at runtime.

GitLab uses CI/CD variables for this purpose.

What Are CI/CD Variables?

Full Control Over Your Kubernetes Infrastructure

Deploy production-ready Kubernetes with full control over nodes, storage, and networking on predictable-cost EU infrastructure with direct access to your resources.

Get

A CI/CD variable is a value that GitLab passes to a job as an environment variable.

For example, you can create the following variable in the project settings:

   DEPLOY_ENV=staging

You can then reference it in .gitlab-ci.yml as follows:

   script:
  - echo "$DEPLOY_ENV"

In a Linux container, the runner sees it as a standard environment variable.

You can use variables to pass:

• API tokens;
• Passwords;
• Deployment URLs;
• Environment names;
• Feature flags;
• Service credentials;
• Other parameters that differ between environments.

In addition to user-defined variables, GitLab automatically provides a large set of predefined variables.

We have already used some of them:

• CI_PROJECT_PATH
• CI_COMMIT_SHORT_SHA

GitLab creates these variables automatically, and they contain information about the current project, commit, pipeline, and job.

User-defined CI/CD variables serve a different purpose: they allow you to pass custom values without storing them directly in the repository.

However, not all variables are the same. The masked and protected attributes are especially important for sensitive data.

How regular variables differ from masked and protected variables

A regular CI/CD variable is simply passed to the job.

If you create:

   APP_ENV=staging

you can use it in a script like a regular environment variable.

This is sufficient for non-secret values.

Secrets are different. If a job accidentally outputs a token using:

   echo "$API_TOKEN"

the value may appear in the job log.

For such cases, use Masked.

GitLab attempts to hide a masked variable in the log by replacing matching values with a mask. This reduces the risk of accidentally exposing a secret in the CI interface.

However, masking should not be considered absolute protection. If a user can modify .gitlab-ci.yml and execute arbitrary code, they may be able to use the supplied secret in other ways. Therefore, permissions to edit the pipeline remain critical.

The second property is Protected.

A protected variable is available only to pipelines that run for protected branches or protected tags.

For example, a deployment token can be marked as protected and used only in pipelines for main, provided that the branch is configured as protected.

In that case, a regular feature branch will not receive the secret, even if such a variable exists in the project settings.

In general, the distinction is as follows:

• Regular variable—simply a value for CI;
• Masked—the value is also hidden from the log;
• Protected—the value is available only to protected refs;
• Masked + protected—a typical configuration for a sensitive production secret.

Now let’s look at how these values get into a container.

How the Runner Receives Variables During a Job

When GitLab assigns a specific job to a Runner, it also creates a set of available CI/CD variables.

This set may include:

• GitLab predefined variables;
• Project variables;
• Group variables;
• Variables for a specific environment;
• Values defined directly in .gitlab-ci.yml.

The Runner receives this set and passes the variables to the job’s environment.

With the Docker executor, they become environment variables inside the container.

For example, if the following variable is defined in GitLab:

   DEPLOY_ENV=staging

you can run the following command within the job:

   echo "$DEPLOY_ENV"

and get:

   staging

Secret values should not be checked this way. If a variable contains a token or password, deliberately printing it to the log is poor practice, even when masking is enabled.

Instead, it is better to check whether the variable has a value.

For example:

   test -n "$API_TOKEN"

The command will complete successfully if the variable is not empty, without displaying its contents.

We will use this approach in practice.

Hands-on: Adding and Verifying a CI/CD Variable

Open the project in GitLab and navigate to the CI/CD variable settings.

Create a test variable:

   Key: DEMO_SECRET
Value: demo-secret-value-123

Enable Masked for this variable.

Leave Protected disabled for now so that the variable is available to pipelines from our current branch, regardless of whether the branch is protected.

Now add a separate job to .gitlab-ci.yml:

   check_variable:
  stage: test
  image: alpine:3.22
  tags:
    - docker
  script:
    - test -n "$DEMO_SECRET"
    - echo "CI/CD variable is available"

Note that the value itself is never printed.

The command:

   test -n "$DEMO_SECRET"

only checks that the variable exists and is not empty.

If everything is configured correctly, the following line will run:

   echo "CI/CD variable is available"

Now stage the changes:

   git add .gitlab-ci.yml

Create a commit:

   git commit -m "Add CI/CD variable check"

Then push it:

   git push

Once the pipeline starts, open the check_variable job.

The following line should appear in the log:

CI/CD variable is available

The value of DEMO_SECRET is not displayed anywhere.

For demonstration purposes, you can also test masking with a temporary test variable:

   script:
  - echo "$DEMO_SECRET"

GitLab should mask the matching value in the job log. It is best to remove this line after testing because real secrets should not be deliberately printed to the console.

The main CI/CD variable options can be summarized in a short table:

The pipeline can now access sensitive parameters separately from the repository, while .gitlab-ci.yml remains suitable for storage in Git.

Next, we will examine the resource cost of this setup. We have already measured the baseline VPS resource usage, so we can now compare it with GitLab while idle and with the server while it is running an actual CI job.

Measuring GitLab and GitLab Runner Resource Usage

Why Resource Usage Should Be Measured While a Real Pipeline Is Running

Measuring the idle state alone is not enough.

If you simply open top immediately after installing GitLab, the system may appear to be under very little load, leading you to conclude that the server has plenty of spare capacity.

However, Runner does not generate a constant load. The load occurs only while a job is running.

The load profile also depends on the pipeline itself. A simple echo command will reveal almost nothing, whereas installing dependencies, running tests, or building an application will noticeably increase CPU, RAM, and disk usage.

It is therefore more useful to compare three states:

1. A clean VPS before GitLab is installed;
2. GitLab is running, but no pipeline is being executed;
3. GitLab and Runner are both running during a CI job.

This allows us to see not only GitLab’s baseline resource usage, but also the additional load generated by Runner.

Metrics to Monitor

For a small VPS, monitoring four groups of metrics is sufficient.

First, CPU.

We are interested in current CPU usage and load average. If the CPUs remain consistently busy during a pipeline, the job is already competing with GitLab for computing resources.

The second metric is RAM.

It is important to check not only used memory but also available memory. Linux actively uses free memory for caching, so a high used value does not necessarily indicate a shortage of RAM.

The third group is disk space.

GitLab gradually accumulates repositories, logs, artifacts, and backups. The Runner also creates working files, Docker layers, and cache data.

Finally, it is useful to examine the processes and containers themselves to determine exactly what is consuming resources.

Ultimately, we will need to monitor:

• CPU usage and load average;
• RAM and swap;
• Free disk space;
• The size of the GitLab and Runner directories;
• The most resource-intensive processes;
• Docker status during a job.

This is enough to provide a basic overview without a dedicated monitoring system.

How Idle Resource Usage Differs from CI Workloads

GitLab continues to run while idle.

Even when no one is using the web interface and no pipelines are running, the database, Redis, Sidekiq, Gitaly, and other components remain active.

As a result, some RAM and CPU resources are constantly in use.

During this time, the runner mostly waits for jobs and typically places a much lighter load on the system.

This changes when a pipeline starts.

The Docker executor must:

• Pull the container image;
• Create the container;
• Prepare the working directory;
• Download the repository;
• Run the job commands;
• Save the cache or artifacts;
• Remove the temporary environment when finished.

As a result, the load is short-lived but significantly higher.

For example, the first use of a container image may place a heavier load on the network and disk because Docker still needs to download its layers. A subsequent job using the same image may complete faster by using the local cache.

Therefore, a single measurement is not always representative. It is better to compare at least the idle state with one actual pipeline run and, if necessary, repeat the test.

How the Runner Affects CPU, RAM, and Disk Usage

The Runner’s resource usage depends primarily on what the job does.

The following tasks place the greatest load on the CPU:

• Compilation;
• Testing;
• Compression;
• Building packages;
• Building containers.

RAM is heavily used by:

• Large dependency trees;
• Parallel tests;
• Building large applications;
• Running multiple jobs simultaneously.

The following place a heavy load on the disk:

• Cloning repositories;
• Docker images;
• Temporary build files;
• Cache;
• Artifacts;
• Package managers.

This is why we previously limited the Runner to:

   concurrent = 1

and the Docker container to:

   cpus = "2"
memory = "4g"

These limits do not make the job less resource-intensive, but they prevent a single pipeline from consuming all the VPS resources unchecked.

When analyzing the results, consider not only the peak values but also GitLab’s behavior. If the web interface remains responsive while the job is running, there is no memory pressure, and the disk has adequate headroom, this setup may be perfectly acceptable for a light workload.

However, if every pipeline noticeably slows down GitLab, that is a strong argument for using a separate VPS for the Runner.

Now let’s collect some real-world metrics.

Hands-on: measuring resource usage before, during, and after the pipeline

We will record it in the table as the baseline:

First, we will capture GitLab’s state while it is idle.

Check memory usage:

   free -h

Check the load:

   uptime

Check free disk space:

   df -h

To view the most resource-intensive processes:

   ps aux --sort=-%mem | head

And separately by CPU usage:

   ps aux --sort=-%cpu | head

For real-time monitoring, it is useful to run:

   top

We can now determine how much disk space GitLab is already using.

For example:

   sudo du -sh /var/opt/gitlab

Check Docker separately:

   sudo docker system df

Next, start a new pipeline.

To make the load more noticeable than with a simple echo command, you can temporarily add a few basic operations to the test job that do not impose a potentially harmful load. However, it is better to take measurements using the actual pipeline that will be used in the article.

While the job is running, repeat:

   free -h
uptime
ps aux --sort=-%cpu | head

And:

   df -h

To view the running containers separately:

   sudo docker ps

To view current Docker resource usage:

   sudo docker stats

This command is particularly useful while a job is running: it shows the CPU and RAM usage of a specific container in near real time.

After the pipeline finishes, collect the metrics again:

   free -h
uptime
df -h

Then check whether the amount of Docker data has changed:

   sudo docker system df

If artifacts or cache were used, you can also check whether the GitLab directories have grown:

   sudo du -sh /var/opt/gitlab

For convenience, here is a summary of the main measurement commands:

After the actual test, all that remains is to enter the measured values here and briefly describe the results: how much RAM usage increased, what happened to the load average, and whether there was a noticeable increase in disk usage.

This will give us a practical answer as to whether a single VPS can run both GitLab and a small Runner at the same time. We can then move on to another critical operational task: backing up GitLab itself.

Configuring GitLab Backups

What a GitLab backup includes

The standard GitLab command backs up the instance’s core data.

The backup may include:

• PostgreSQL database;
• Git repositories;
• project and group wikis;
• uploads and attachments;
• CI job logs;
• CI artifacts;
• Git LFS objects;
• GitLab Pages;
• Package Registry data;
• Terraform state;
• project-level Secure Files;
• Container Registry data, if it is stored locally and is compatible with the configuration in use.

In other words, a backup covers more than just source code.

This is important because a significant portion of GitLab’s state resides in PostgreSQL. It stores users, projects, permissions, issues, merge requests, CI/CD metadata, and many other objects.

In simplified terms, a backup combines several types of data:

Therefore, copying only the repositories directory is not a substitute for a complete backup.

However, gitlab-backup itself does not back up the entire server.

What Is Not Automatically Included in a Standard Backup

There are several important exceptions.

A standard GitLab backup does not include:

• /etc/gitlab;
• gitlab.rb;
• gitlab-secrets.json;
• TLS certificates and private keys;
• SSH host keys;
• other system files;
• Redis state, including current Sidekiq jobs;
• data stored in external object storage in a standard Linux package deployment.

The last point is particularly important.

If artifacts, uploads, LFS, or the Registry have already been moved to S3-compatible object storage, running only:

   sudo gitlab-backup create

may not be sufficient. The object storage itself must be backed up separately.

For our small single-node installation, we can continue using local storage for now. However, even in this setup, there are two files without which recovery can become a serious problem:

   /etc/gitlab/gitlab.rb
/etc/gitlab/gitlab-secrets.json

We will discuss them separately.

Why the Configuration and Secrets Must Be Backed Up Separately

gitlab.rb contains the configuration for the GitLab instance itself.

This is where parameters such as the following are defined:

   external_url "https://gitlab.example.com"

as well as settings for backups, NGINX, SMTP, storage, and other components.

Without this file, the data can still be restored, but the server configuration will have to be recreated.

More important still: /etc/gitlab/gitlab-secrets.json

This file contains the cryptographic secrets that GitLab uses to work with encrypted data.

For example, the database may contain encrypted values. If you restore the database but lose the corresponding keys, the application will no longer be able to decrypt some of this data correctly.

Therefore, the following situation:

gitlab_backup.tar is available

gitlab-secrets.json is missing

may be much worse than it appears.

Secrets are intentionally excluded from the main backup archive. Storing encrypted data and the key used to decrypt it in the same location is a bad idea.

Therefore, we create two independent sets:

• a backup of GitLab data;
• a secure copy of the configuration and secrets.

Ideally, they should be stored separately from each other.

Where GitLab Stores Backups

For the Linux package, the default backup directory is:

   /var/opt/gitlab/backups

The generated archive will have a name similar to this:

   <backup-id>_gitlab_backup.tar

You can check the contents of the directory with the following command:

   sudo ls -lh /var/opt/gitlab/backups

You can change the storage location in /etc/gitlab/gitlab.rb.

For example:

   gitlab_rails['backup_path'] = '/var/opt/gitlab/backups'

After changing the configuration, run:

   sudo gitlab-ctl reconfigure

However, keeping the only backup on the same VPS is a poor strategy.

If the server and its disk are lost, both GitLab and its backup will be lost.

Therefore, it is best to treat the local directory as a staging location from which the archive is subsequently transferred to another storage system.

How to Plan Backup Storage

A simple strategy is sufficient for a small GitLab instance.

For example:

• Daily backups;
• Several recent daily backups;
• One or more older reference backups;
• Separate external storage;
• A separate, secure copy of /etc/gitlab.

This could be:

• A second server;
• A NAS;
• S3-compatible object storage;
• A separate backup volume.

The guiding principle here is simple: the backup must not disappear along with the GitLab VPS.

You should also consider the volume of data. If GitLab occupies tens of gigabytes, retaining daily archives indefinitely will quickly fill the storage.

For a Linux package installation, you can configure how long old local backups are retained.

For example:

   gitlab_rails['backup_keep_time'] = 604800

604800 seconds is 7 days.

This helps automatically remove old local archives, but it does not replace a retention policy for external storage.

One more important point: successfully creating a .tar file does not prove that the backup can actually be restored. In the next chapter, we will perform a full validation by restoring it.

For now, we will create the archive.

Hands-on: Creating a Backup

Before you begin, check the available disk space:

   df -h

A backup may temporarily require a significant amount of disk space, especially if the repositories and artifacts have already grown large.

Now run the standard command:

   sudo gitlab-backup create

GitLab will process each component in sequence and create the final archive.

Once the process is complete, check the directory:

   sudo ls -lh /var/opt/gitlab/backups

It should contain a file matching this pattern:

   <backup-id>_gitlab_backup.tar

Now save the configuration separately.

You can create a protected archive of /etc/gitlab:

   sudo tar -czf /root/gitlab-config-$(date +%F).tar.gz /etc/gitlab

Then verify that it was created:

   sudo ls -lh /root/gitlab-config-*.tar.gz

This archive contains sensitive data, including secrets and potentially TLS private keys. Do not store it in a public repository or a regular shared folder.

Better yet, transfer both the main backup and the configuration archive to separate secure storage after creating them.

For our installation, the backup plan is as follows:

At this point, the backup has been created, but we have verified only the first half of the process.

Restoring GitLab from a Backup

Now we need to test the second, more important half of the process— whether GitLab can actually be restored from it.

The restore process itself is technically straightforward, but there are several critical considerations. You cannot simply take any .tar file, restore it over an arbitrary GitLab version, and expect everything to work. Before restoring, you need to verify the version, restore the configuration and secrets, stop certain services, and understand what will happen to the existing data.

Why the GitLab Version Matters During Recovery

A GitLab backup is tied not only to the data but also to the application version.

For a successful restore, the target instance must be running the same GitLab version and editionthat were used to create the backup.

For example, if the backup was created on:

GitLab CE 18.x.y

it must first be restored on the same GitLab CE 18.x.y version. Once the restore is successful, upgrade the instance using the standard procedure.

Do not assume that the following scenario will work: old backup → much newer GitLab version → direct restore

This is because the database schema, migrations, data formats, and internal application logic change between versions.

Therefore, it is useful to save the version information along with the backup:

   sudo gitlab-rake gitlab:env:info

or at least:

   sudo gitlab-rake gitlab:env:info | grep "GitLab information" -A 10

This greatly simplifies recovery after a complete VPS failure.

However, having the correct version is not enough. Several other items must also be restored before running the restore operation.

What to Prepare Before the Restore

The restore requires three main components:

• A clean installation of the required GitLab version;
• The primary backup archive;
• The saved configuration from /etc/gitlab.

The following file is particularly important: /etc/gitlab/gitlab-secrets.json

If it was backed up, restore it before fully validating the restored GitLab instance.

It is also advisable to restore: /etc/gitlab/gitlab.rb

to reinstate the same external_url, storage, SMTP, backup, and other instance settings rather than configuring them manually.

If the TLS certificates and SSH host keys were backed up separately, they should also be restored to their original locations.

Copy the backup archive to the GitLab backup directory: /var/opt/gitlab/backups

For example:

   sudo cp 1760000000_2026_10_02_18.x.y_gitlab_backup.tar \
  /var/opt/gitlab/backups/

The actual file name will depend on the GitLab version and the time the backup was created.

Verify it:

   sudo ls -lh /var/opt/gitlab/backups

GitLab must be able to read this file, so if it was copied from another server, check its ownership and permissions.

Once the files are ready, you can proceed to stop the services.

Why Some Services Are Stopped Before a Restore

During a restore, GitLab modifies the database and restores data from the backup.

If users continue to push commits, create issues, or run pipelines during this process, the current state may become mixed with the state being restored.

Data writes must therefore be temporarily suspended.

However, you do not need to shut down the entire GitLab instance before starting the restore. The internal components required for the restore process, particularly PostgreSQL, must remain available.

Application processes are typically stopped before a restore:

   sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq

Then check the status:

   sudo gitlab-ctl status

PostgreSQL and the other required services must remain running.

The rationale is straightforward: user-initiated and background operations are temporarily suspended, while the infrastructure required by the restore utility remains available.

The instance state can then be modified more safely.

What Happens to Current Data During a Restore

A restore does not merge the backup with the current GitLab instance.

It returns the instance to the state captured in the backup.

This means that data created after the backup was made may be lost.

Consider the following example:

1. 12:00 — Backup created
2. 13:00 — New commit created
3. 14:00 — New issue created
4. 15:00 — Restore performed using the backup from 12:00

After the restore, GitLab returns to the state captured in the backup. The commit and issue created later are not added automatically.

This is why a restore should not be treated as a harmless diagnostic command on a live production instance.

Before performing a restore, administrators typically:

• Suspend user activity;
• Verify that the correct backup has been selected;
• Create an additional backup of the current state if necessary;
• Only then start the restore.

The risk is low in a test environment, but it is still worth following the correct procedure from the outset.

Hands-on: Performing the Restore

First, check which backup is available:

   sudo ls -lh /var/opt/gitlab/backups

Assume the file is named:

   1760000000_2026_10_02_18.x.y_gitlab_backup.tar

To restore GitLab, you typically need the backup ID without the _gitlab_backup.tar suffix.

Therefore, use:

   1760000000_2026_10_02_18.x.y

Before restoring, verify once again that the GitLab version matches the backup version.

Then stop Puma and Sidekiq:

   sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq

Do not worry: after the restore, we will run gitlab-ctl restart, and the previously stopped Puma and Sidekiq services will start again along with the other GitLab components.

Check the service status:

   sudo gitlab-ctl status

Now start the restore:

   sudo gitlab-backup restore \
  BACKUP=1760000000_2026_10_02_18.x.y

GitLab will warn you that the current database data will be replaced. Carefully verify the selected backup and confirm the operation.

Once the restore is complete, restore the configuration if it was lost along with the old VPS.

For example, from an archive saved in advance:

   sudo tar -xzf /root/gitlab-config-2026-10-02.tar.gz -C /

Before extracting an archive this way, make sure you understand its contents: it must contain the expected /etc/gitlab directory, not unknown files from a third-party source.

Check that the main files are present:

   sudo ls -l /etc/gitlab/gitlab.rb
sudo ls -l /etc/gitlab/gitlab-secrets.json

Now apply the configuration:

   sudo gitlab-ctl reconfigure

Then restart GitLab:

   sudo gitlab-ctl restart

Then check the status:

   sudo gitlab-ctl status

It is also useful to run the built-in checks:

   sudo gitlab-rake gitlab:check SANITIZE=true

The restore workflow is therefore as follows:

The main commands can be summarized in a short table:

The command-level restore is now complete, but this is not yet enough to declare the procedure successful.

In the next section, we will verify the GitLab contents: confirm that the projects and repositories have been restored, the Runner is still available, and a new pipeline actually starts after the restore.

Verifying GitLab After Restoration

What to check first

Let’s start with the instance itself.

Open GitLab at its usual URL: https://gitlab.example.com

Verify that:

• The page loads over HTTPS;
• You can sign in as an existing user;
• Projects are displayed;
• Groups and settings are intact;
• There are no obvious UI errors.

Next, check the status of the internal services:

   sudo gitlab-ctl status

If any component is not running, stop at this point and check it separately.

It is also useful to run:

   sudo gitlab-rake gitlab:check SANITIZE=true

This command helps quickly identify issues with the configuration, permissions, and internal components.

If the basic GitLab functionality is working, proceed to the repositories.

How to Verify That the Repositories Were Restored

The presence of a project in the web interface alone is not sufficient.

A project may appear in the database while its repository is corrupted or unavailable. Therefore, it is best to test Git operations separately.

Let’s open our demo-app project and check:

• Commit history;
• Branches;
• Files;
• The last commit before the backup.

Then, on a workstation, try running:

   git clone [email protected]:user/demo-app.git

If a local copy already exists:

   cd demo-app
git fetch

You can then check the commit history:

   git log --oneline -5

It should include the commits that existed when the backup was created.

It is also useful to make a test change:

   echo "restore check" >> README.md
git add README.md
git commit -m "Check repository after restore"
git push

If the push succeeds, GitLab can not only read the restored repository but also accept new changes after the restore.

You can now verify the CI functionality.

What to Check on the Runner After a Restore

GitLab Runner is typically a separate service with its own local configuration:

   /etc/gitlab-runner/config.toml

Therefore, restoring GitLab does not automatically restore the Runner files.

If the Runner is still on the same VPS and its configuration has not been deleted, the service may continue to work as before.

Check its status:

   sudo systemctl is-active gitlab-runner

Then run:

   sudo gitlab-runner verify

Next, open Settings → CI/CD → Runners and make sure that demo-docker-runner is shown as available.

If the Runner is online, that is a good sign, but the definitive test is a new job.

This raises an important question: does the Runner always need to be registered again after a restore?

Is it necessary to re-register the Runner?

Not necessarily.

If GitLab Runner retained its configuration file:

   /etc/gitlab-runner/config.toml

and the restored GitLab instance contains the previous Runner record and its associated data, the existing registration may continue to work.

Therefore, when restoring the same instance, you should generally check the existing Runner before registering a new one.

Re-registration is required if:

• The Runner VPS was lost;
• The config.toml file was not preserved;
• The Runner was deleted from GitLab;
• The authentication token is no longer valid;
• The GitLab instance is being changed;
• The existing registration is intentionally being replaced.

This is important because unnecessary re-registration may simply create a duplicate Runner in the UI.

The correct sequence is therefore as follows:

After that, all that remains is to test the entire pipeline.

Post-Restore Verification Commands

Let’s start with the GitLab services:

   sudo gitlab-ctl status

Check the internal state:

   sudo gitlab-rake gitlab:check SANITIZE=true

View the environment information:

   sudo gitlab-rake gitlab:env:info

Check the Runner:

   sudo systemctl is-active gitlab-runner

and:

   sudo gitlab-runner verify

Now create a new commit in demo-app:

   echo "CI after restore" >> README.md
git add README.md
git commit -m "Test CI after restore"
git push

After pushing, go to Build → Pipelines.

The new pipeline should follow the same sequence as it did before the restore.

If the stages complete with a passed status, this confirms several things at once:

• GitLab accepts pushes;
• The repository is accessible;
• The CI configuration can be read;
• The Runner is connected to GitLab;
• The Docker executor starts;
• Jobs run after the restore.

In summary, the entire verification process can be divided into four levels:

If all four levels are working, the restore can be considered verified not only at the restore command level, but also under actual operating conditions.

We now have a working backup and restore workflow. The next section will cover operational issues: why a CI job may become stuck in the pending state, why the Runner sometimes fails to pick up a job, and how to quickly isolate the cause.

Diagnosing Stuck CI Jobs

What Does It Mean When a Job Is Pending?

The pending status means that GitLab has already created the job, but execution has not yet started.

In other words, .gitlab-ci.yml has been processed and the pipeline exists, but a runner has not yet picked up the job.

It is normal for a job to remain pending briefly. The runner may be busy with a previous job or may simply not have requested a new job yet.

It becomes a problem when the job remains in this state for a long time.

In that case, check the following:

• Whether there is an active runner at all;
• Whether it is available to this project;
• Whether the tags match;
• Whether the runner is allowed to run untagged jobs;
• Whether the concurrency limit has been reached;
• Whether the runner is paused;
• Whether any restrictions apply to protected branches.

For example, if our runner is already executing one job and config.toml contains the following setting:

   concurrent = 1

A second job may have to wait until the runner becomes available. This is not an error—it is simply waiting in the queue.

However, if a runner is available but the job still does not start, you need to investigate the cause further.

Why a Runner May Not Pick Up a Job

The fact that GitLab displays a Runner in the UI does not guarantee that it can execute a specific job.

One of the first things to check is the status of the local service:

   sudo systemctl is-active gitlab-runner

If the service is stopped, GitLab cannot assign a new job to it.

Next, verify the registration:

   sudo gitlab-runner verify

If this command returns a connection or authentication error, the issue lies between the Runner and GitLab.

A Runner may also be available but busy with another job. This is particularly relevant to our configuration because we deliberately limited concurrency to one job.

You can check the current processes with:

   ps aux | grep gitlab-runner

and Docker containers with:

   sudo docker ps

If a job is already running, the next one will naturally have to wait.

Another group of possible causes relates not to the Runner service itself, but to the criteria GitLab uses when selecting a Runner.

How Tags Can Leave a Job Without a Matching Runner

High-Performance Virtual Machines

Deploy in seconds with full root access, reserved resources, built-in high availability, and predictable monthly pricing.

Get

Tags are one of the most common reasons a job remains pending indefinitely.

In our environment, the Runner was assigned the following tag:

   docker

The job uses:

   tags:
  - docker

These values must match.

If you accidentally specify:

   tags:
  - docker-runner

or:

   tags:
  - linux

and the Runner does not have the corresponding tag, GitLab cannot assign the job to it.

Another common case is when multiple tags are specified:

   tags:
  - docker
  - linux

The Runner must match the entire required set.

If it only has:

   docker

the job will not be assigned to it.

Therefore, if a job is stuck in the pending state, open the Runner settings in GitLab and compare the list of tags directly with those specified in .gitlab-ci.yml.

The reverse can also happen: the job has no tags, but the Runner is not allowed to run untagged jobs. In this case, the job will also have no Runner available to execute it.

So the first diagnostic rule here is simple: if the Job has not started running yet, first check whether the Runner is a match, rather than the commands in the script.

What Is a Stuck Job?

The term “stuck job” usually refers to a job that cannot be assigned to a suitable runner.

In other words, the job exists, but the conditions required to run it are not met.

GitLab may explicitly indicate that a job is stuck because no runner matches its tags or other parameters.

Common causes include:

• All runners are offline;
• The runner is paused;
• No runner has the required tags;
• The runner is not permitted to run untagged jobs;
• The runner is not available to this project;
• The job is associated with a protected branch, but the runner is not configured accordingly;
• All suitable runners are busy;
• The concurrency limit has been reached.

It is useful to distinguish between two levels here:

• Stuck/pending before start — an issue with Runner selection or availability;
• Running, but nothing is happening — The Job has already been assigned a Runner, so the issue must be investigated within the execution process.

This significantly speeds up troubleshooting.

Why a Job Can Hang After It Starts

If the status has changed to running, GitLab has already found a Runner and assigned the Job to it.

This means that tags, registration, and basic connectivity to GitLab are most likely working correctly.

The problem may now be within the Job itself.

For example, a command may be waiting for user input:

   some-command

and the program opens an interactive prompt. In CI, no one will press Y, so the Job may remain stuck indefinitely.

A similar situation can occur with package managers if non-interactive mode is not enabled.

Another common cause is a network operation that does not complete.

The Job may be attempting to:

• Download dependencies;
• Connect to an API;
• Connect to a database;
• Retrieve a package from an external registry;
• Resolve a DNS name.

If the external service does not respond, the command may take a long time to time out.

The Docker executor may also encounter specific issues:

• The image fails to download;
• The Docker daemon is unavailable;
• There is insufficient disk space;
• The container has reached its memory limit;
• Filesystem I/O has stalled;
• The registry requires authentication.

For example, you can check the Docker service status as follows:

   sudo systemctl is-active docker

Available disk space:

   df -h

Memory:

   free -h

Running containers:

   sudo docker ps

The Job log remains the primary source of information. The last line often shows the exact operation where execution stopped.

If the Job hangs at: Installing dependencies…

you should troubleshoot the package manager and network rather than the Runner registration.

Runner and CI Diagnostic Commands

Start with the Runner itself:

   sudo systemctl status gitlab-runner

For a quick check:

   sudo systemctl is-active gitlab-runner

Now check its registration:

   sudo gitlab-runner verify

If the service is running, review its latest logs:

   sudo journalctl -u gitlab-runner -n 100 --no-pager

For real-time monitoring:

   sudo journalctl -u gitlab-runner -f

When using the Docker executor, check Docker separately:

   sudo systemctl is-active docker

Running containers:

   sudo docker ps

All containers, including stopped ones:

   sudo docker ps -a

Check the available resources:

   free -h
df -h
uptime

Also check the Runner configuration:

   sudo cat /etc/gitlab-runner/config.toml

Do not copy the authentication token from the output into screenshots or documentation.

In GitLab, check the following:

• Whether the Runner is online or offline;
• Whether it is paused;
• Its tags;
• Whether untagged jobs are allowed;
• Whether it is available to the required project;
• Whether its protected status is appropriate;
• Which Runner is assigned to the affected job.

For quick troubleshooting, work from the symptoms:

The key is not to troubleshoot CI haphazardly. First, determine the job’s status.

If it is still pending, check GitLab → Runner → tags → restrictions.

If it is already running, check the Runner → executor → container → specific command in the job log.

This makes it much faster to isolate the problem.

What to Do If a Runner Is Offline

In the previous section, we covered jobs that remain pending or hang during execution. Now let’s look at a more straightforward case: GitLab no longer considers the Runner available and shows it as offline.

In this case, the issue is usually not in .gitlab-ci.yml but at a lower level—in the Runner service itself, its connection to GitLab, or its registration settings. Therefore, it is best to start troubleshooting with the local process on the VPS rather than the pipeline.

Why GitLab Can No Longer See the Runner

GitLab considers a Runner available as long as it regularly contacts the server and requests new jobs.

If this communication stops, the Runner will eventually be shown as unavailable.

This can happen for a variety of reasons:

• The gitlab-runner service has stopped;
• The VPS restarted, but the service did not start;
• The network connection was lost;
• The Runner can no longer resolve the GitLab DNS name;
• A TLS error occurred;
• The GitLab URL changed;
• The authentication token became invalid;
• The config.toml file was corrupted or deleted;
• A firewall or another network setting is blocking the connection.

GitLab and the Runner may even be hosted on the same VPS but still communicate through the specified URL:

   url = "https://gitlab.example.com"

Therefore, a working GitLab web interface does not guarantee that the Runner can establish a connection from its side.

The first thing to check is whether the local service is running at all.

How to Check the systemd Service

The Runner is installed as a systemd service, so start with this simple command:

   sudo systemctl status gitlab-runner

If everything is working correctly, the service should have the following status:

   active (running)

For a quick check without detailed output, use:

   sudo systemctl is-active gitlab-runner

If the result is inactive or failed:

The Runner will no longer be able to receive new jobs.

Try starting it:

   sudo systemctl start gitlab-runner

Then check the status again:

   sudo systemctl status gitlab-runner

To configure the Runner to start automatically after a reboot:

   sudo systemctl enable gitlab-runner

If the service starts and then immediately crashes, simply restarting it is unlikely to help. Check the logs to identify the cause:

   sudo journalctl -u gitlab-runner -n 100 --no-pager

To monitor the logs in real time:

   sudo journalctl -u gitlab-runner -f

The logs may reveal errors reading the configuration, network issues, or other reasons why the process terminated.

If systemd reports the service as active but GitLab still considers the Runner offline, the next step is to check the connection between them.

How to Check Connectivity Between the Runner and GitLab

To verify registered Runners, use:

   sudo gitlab-runner verify

This command checks whether the local Runner can connect to GitLab and authenticate.

If everything is working correctly, the corresponding Runner should pass verification.

If an error occurs, you need to determine which layer the problem is at.

If you suspect a DNS issue, check the domain:

   getent hosts gitlab.example.com

or:

   dig +short gitlab.example.com

Next, check HTTPS:

   curl -I https://gitlab.example.com

If curl cannot establish a TLS connection, the Runner will most likely encounter the same problem.

You can also check whether GitLab is accessible from the VPS itself:

   curl -I https://gitlab.example.com/users/sign_in

If DNS and HTTPS are working, check the local configuration:

   sudo nano /etc/gitlab-runner/config.toml

In the Runner section, first check:

   
[[runners]]
  name = "demo-docker-runner"
  url = "https://gitlab.example.com"
  executor = "docker"

Pay particular attention to the url setting.

Even a minor issue, such as an outdated hostname or GitLab moving to a different domain, can completely break connectivity.

If the URL is correct, the remaining items to check are the registration and authentication token.

What Happens After Changing the URL or Token

It is useful to distinguish between two different types of changes.

If the GitLab address itself changes, for example, from:

to:

the existing Runner may continue connecting to the address specified in its config.toml file.

In this case, update:

   url = "https://gitlab.example.com"

and restart the service:

   sudo systemctl restart gitlab-runner

The authentication token is handled more strictly.

The Runner uses it to confirm its registration with GitLab. If the token has been revoked or the Runner has been recreated in the UI, the old value will no longer work.

In this case, repeatedly running the following command is not enough:

   sudo systemctl restart gitlab-runner

The service will start, but the connection to GitLab will not be restored.

If a new authentication token has been issued, you must either update the registration correctly or remove the old Runner configuration and register it again.

For example, you can first view the current registrations:

   sudo gitlab-runner list

If the old Runner is no longer needed, you can unregister it:

   sudo gitlab-runner unregister --name "demo-docker-runner"

Then register the Runner again:

   sudo gitlab-runner register

and specify the following again:

• GitLab URL;
• New authentication token;
• Executor;
• Runner settings.

However, not every error requires re-registration. If the issue is simply a stopped service or a DNS problem, recreating the Runner is unnecessary.

Therefore, always use verify to check the current registration first.

Runner Recovery Commands

First, check the service:

   sudo systemctl status gitlab-runner

If it is not running:

   sudo systemctl restart gitlab-runner

Verify the registration:

   sudo gitlab-runner verify

If the issue persists, check the logs:

   sudo journalctl -u gitlab-runner -n 100 --no-pager

Check the GitLab URL:

   curl -I https://gitlab.example.com

Then check the configuration:

   sudo nano /etc/gitlab-runner/config.toml

If necessary, view the list of registered Runners:

   sudo gitlab-runner list

If the registration is no longer valid, remove it:

   sudo gitlab-runner unregister --name "demo-docker-runner"

and register the Runner again:

   sudo gitlab-runner register

After making changes, restart the service:

   sudo systemctl restart gitlab-runner

and perform a final check:

   sudo systemctl is-active gitlab-runner
sudo gitlab-runner verify

The main sequence is useful to keep as a quick reference:

If the Runner appears online again afterward, it is useful to run a small test pipeline. This checks not only the heartbeat between the Runner and GitLab, but also the complete path through the Docker executor to job execution.

Next, we will examine the opposite scenario: the Runner is online and the job starts, but the pipeline itself fails. In this case, you need to distinguish an infrastructure issue from an error in the CI commands.

Troubleshooting a Pipeline That Starts but Fails

How a Pipeline Error Differs from a Runner Issue

If a job does not start at all and remains pending, check the Runner, tags, registration, and concurrency settings.

If a job transitions to running and then to failed, GitLab found a Runner, the Runner received the job, and the executor began executing it.

This means the basic chain up to the Runner is working correctly.

The next step is to troubleshoot the job itself.

For example, the following pipeline:

   test:
  stage: test
  image: alpine:3.22
  tags:
    - docker
  script:
    - ./run-tests.sh

may fail for many different reasons:

• The run-tests.sh file does not exist;
• The file does not have execute permission;
• An error occurred within the script;
• A required package is missing;
• Command not found;
• The system ran out of memory;
• Network access is unavailable;
• A secret variable was not passed to the job.

Therefore, the failed status alone reveals very little. You need the job log.

Where to Find the Job Log

Go to Build → Pipelines → the relevant pipeline → the relevant job.

The complete job execution log is available there.

The Runner typically displays several stages in sequence:

• Preparing the executor;
• Fetching the source code;
• Restoring the cache or artifacts;
• Executing the script;
• Uploading artifacts;
• Cleanup.

We are primarily interested in the last successful operation and the first error after it.

For example:

   $ ./run-tests.sh
/bin/sh: ./run-tests.sh: not found
ERROR: Job failed: exit code 127

The problem can be identified almost immediately: the Runner is working, and the container has started, but the required command or file is unavailable.

Another example:

   $ npm test
npm: not found

This is not a GitLab issue. The selected container image simply does not include npm.

This is why you should read the job log from top to bottom only until you reach the first actual failure, rather than trying to analyze every Runner status message.

How to Distinguish Command Errors from Environment Issues

The easiest way to diagnose the issue is to see exactly where the Job fails.

If the command runs but returns a non-zero exit code:

   Tests failed
exit code 1

the issue is most likely in the application or test logic.

If you get:

   command not found

this indicates an environment issue.

For example:

   python: not found

this means that the image being used does not include Python.

If the error looks like this:

   Permission denied

check the permissions.

If you get:

   connection refused

or:

   Could not resolve host

the issue is more likely related to the network or an external service.

Errors can be broadly categorized as follows:

This classification often helps identify the root cause faster than immediately trying to change the Runner.

Why a Pipeline May Work Locally but Fail on a Runner

This is a very common situation.

A developer’s computer may already have:

• The required version of Node.js;
• Python;
• Java;
• Package managers;
• Environment variables;
• A local database;
• SSH keys;
• Credentials;
• System libraries.

In contrast, a Runner using a Docker container starts in a much cleaner environment.

For example, the following command may work locally:

npm test

because Node.js is already installed.

However, the pipeline uses:

   image: alpine:3.22

which does not include npm at all.

The result is:

   npm: not found

This does not mean that the Runner is malfunctioning. The local and CI environments are simply different.

Environment variables are another common issue.

Locally, the application may read:

   .env

but this file is not present in CI.

As a result, tests begin to fail because credentials or a URL are missing.

The same applies to the working directory. A developer may manually run a command from the correct directory, while the Runner starts the job in the repository root.

A well-designed CI setup should therefore define its environment explicitly rather than assuming that something is already installed on the server.

Hands-on: A basic troubleshooting workflow

Suppose a job finishes with a failed status.

First, open the job log and look for the first meaningful error.

If you see:

   command not found

check the image in .gitlab-ci.yml.

For example:

   image: node:22-alpine

instead of:

   image: alpine:3.22

if the job actually requires Node.js.

If the issue is related to a file:

   No such file or directory

check the repository contents:

   git ls-files

and the path specified in the CI configuration.

If you see:

   Permission denied

check the file permissions:

   ls -l

For a shell script, run the following commands if necessary:

   chmod +x run-tests.sh
git add run-tests.sh
git commit -m "Fix script permissions"
git push

If the job fails because of a variable, check whether it has a value without printing the secret itself:

   script:
  - test -n "$API_TOKEN"

If the error is network-related, you can temporarily test DNS and HTTPS directly within the job:

   script:
  - wget -S --spider https://example.com

or use a suitable image with curl.

If you suspect a resource issue, check the VPS:

   free -h
df -h
uptime

and Docker:

   sudo docker stats

as well as:

   sudo docker ps -a

If the container terminates unexpectedly, it is useful to check the Runner logs:

   sudo journalctl -u gitlab-runner -n 100 --no-pager

The resulting troubleshooting workflow is fairly short:

1. Check the job status;
2. Open the job log;
3. Find the first actual error;
4. Identify the layer where the error occurred;
5. Check the image, variables, files, network, or resources;
6. Fix the issue;
7. Run the pipeline again.

For quick reference when identifying the cause, keep the following table handy:

If the Runner is online and the job starts, the issue can usually be isolated using the job log without touching the Runner service itself.

This covers the main CI troubleshooting scenarios. The next logical step is to verify the entire installation from end to end and ensure that GitLab, the repository, Runner, pipeline, backup, and restore processes work as a single system.

Validating the Entire Deployment

Start with GitLab itself and its system services:

If this layer is working, the core platform is accessible to users and ready to handle Git operations.

Next, check the CI components. Do not rely solely on the online status shown in the GitLab interface: the Runner must actually pick up and successfully execute a new job.

The most useful test here is a small new commit. It exercises nearly the entire workflow at once: GitLab accepts the push and creates a pipeline, the Runner picks up the job, the Docker executor starts a container, and the result is returned to the interface.

However, even a fully functional CI setup does not make the deployment reliable. The final layer is to verify that GitLab can be restored after the loss of data or the VPS itself.

If all three layers pass these checks, the infrastructure can be treated as an integrated system rather than simply a collection of running services.

The complete workflow is as follows:

GitLab accepts and stores changes, the pipeline defines the required automation, the Runner picks up a matching job, and the executor creates the environment in which the commands run.

Two other important mechanisms support the core workflow: resource controls and backups. Resource controls prevent CI from silently consuming the resources required by GitLab itself on the VPS, while backups make it possible to restore the system after a major failure.

The entire deployment has now been validated. The remaining work is less about launching new components and more about operating the existing ones correctly: restrict Runner permissions, protect CI/CD secrets, monitor disk space, update GitLab, and regularly verify not only that backups are created, but also that they can be restored.

How to Use GitLab and GitLab Runner Securely in Production

Do Not Run Untrusted Code on a Privileged Runner

A Runner executes the commands defined in .gitlab-ci.yml.

Therefore, any user who can modify the CI configuration and start a pipeline can potentially affect the Runner environment.

The Docker executor’s privileged mode is particularly dangerous: privileged = true

It significantly expands the container’s capabilities and reduces its isolation from the host system.

If this mode is not strictly required, leave it disabled: privileged = false

Our Runner is configured this way.

Host volumes require even greater caution.

For example, mounting the Docker socket:

   /var/run/docker.sock

inside the CI container effectively gives the job extensive control over Docker on the host. This is a particularly serious concern on a server that also runs GitLab itself.

Therefore, untrusted projects should not be run on a Runner that has elevated privileges or access to critical resources on the primary VPS.

As CI becomes more complex, it is safer to move such Runners to separate machines.

Restrict access to CI/CD variables

CI/CD variables often contain the most sensitive data in a pipeline:

• API tokens;
• Passwords;
• Deployment keys;
• Registry credentials;
• Access tokens.

Therefore, rather than simply creating a secret, determine from the outset which pipeline actually needs it.

Sensitive values should be masked and, where necessary, restricted to a protected scope.

For example, a production token should only be made available to pipelines running on a protected branch:

   main

while feature branches should not have access to it at all.

This is particularly important for projects with multiple developers. If a user can modify .gitlab-ci.yml and a job automatically receives a production secret, that secret can no longer be considered properly isolated.

In other words, protecting CI/CD variables involves not only selecting the Masked checkbox, but also controlling who can run code with access to the variable.

Do Not Store Runner Tokens or Secrets in the Repository

Secrets must not end up in Git history.

This applies not only to deployment passwords, but also to the infrastructure’s own sensitive data:

• Runner authentication tokens;
• Personal access tokens;
• API keys;
• Private SSH keys;
• The contents of gitlab-secrets.json;
• Database passwords;
• Private TLS keys.

For example, you must not add the actual file to the repository:

   /etc/gitlab-runner/config.toml

because the actual file contains Runner authentication data.

The same applies to:

   /etc/gitlab/gitlab-secrets.json

Even a private repository is not an appropriate place for such files.

If a secret is accidentally committed to Git, simply deleting the line is not enough. The value must be considered compromised, revoked, and replaced with a newly issued one.

Separate protected branches from regular branches

Not all branches are equally important.

A regular feature branch is used for development and experimentation. The main branch or another production branch may trigger deployments and access production secrets.

Their permissions should therefore differ as well.

Protected branches let you restrict:

• Who can push;
• Who can merge;
• Which CI/CD variables are available;
• Which runners can execute the corresponding jobs.

For example, a developer can create feature branches and run tests in them, but deployment to production occurs only after changes are merged into the protected main branch.

CI thus becomes more than just automation: it serves as an additional control layer between routine development and production.

Monitor Free Disk Space

For GitLab, free disk space is just as important a resource as RAM and CPU.

Repositories grow. Pipelines create artifacts. Docker accumulates images and layers. GitLab writes logs. Backups also require storage space.

The problem can develop gradually:

• Today: Disk usage: 45%
• A few months later: Disk usage: 82%
• Then, after a large backup or several resource-intensive pipelines, the server’s disk may suddenly be almost full.

You should therefore check disk usage regularly—or, better yet, configure automated monitoring:

   df -h

Size of GitLab data:

   sudo du -sh /var/opt/gitlab

and Docker disk usage:

   sudo docker system df

In production, it is better to take action in advance rather than wait until the disk is 100% full.

In addition to removing old data, it makes sense to configure retention policies for artifacts, packages, and backups based on the project’s actual requirements.

Store Backups Outside the GitLab VPS

A local backup is convenient, but it does not protect against the loss or compromise of the server itself.

If GitLab and its backups are stored on the same virtual disk, damage to or deletion of the VPS will destroy both.

Therefore, after running:

   sudo gitlab-backup create

the archive must be transferred to external storage.

Suitable options include:

• A dedicated backup server;
• A NAS;
• A separate volume;
• S3-compatible object storage.

The /etc/gitlab directory, including gitlab-secrets.json, must also be backed up and stored with particular care because it contains sensitive data.

A reliable backup must survive the loss of the primary GitLab VPS.

Test restores, not just backup creation

A successful backup command does not necessarily mean that disaster recovery is truly ready.

Some issues may only become apparent during a restore:

• The gitlab-secrets.json file is missing;
• The GitLab version was not recorded;
• The archive is corrupted;
• The backup is not in the expected location;
• The configuration backup is outdated;
• The repositories were not fully restored;
• The runner no longer works after the restore.

You should therefore periodically perform a test restore in an isolated environment.

Afterward, verify at least the following:

• Signing in to GitLab;
• Project availability;
• Git clone;
• Git push;
• The runner;
• A new pipeline.

If all these checks succeed, the backup has been validated in practice.

Update GitLab and Runner Regularly

Self-managed GitLab requires ongoing maintenance.

Updates include bug fixes, security fixes, and new versions of internal components. Runner is developed separately, so it should not be left on an outdated version for years.

However, GitLab should not be upgraded without proper preparation.

Before upgrading, it is a good idea to:

1. Check the current version;
2. Review the supported upgrade path;
3. Create a fresh backup;
4. Back up /etc/gitlab;
5. Make sure there is enough disk space;
6. Perform the upgrade;
7. Check the services and pipeline.

You can view the current GitLab environment information as follows:

   sudo gitlab-rake gitlab:env:info

To check the Runner version:

   gitlab-runner --version

After the upgrade, run the following checks again:

   sudo gitlab-ctl status
sudo gitlab-rake gitlab:check SANITIZE=true

and run a test pipeline.

This verifies the upgrade not only by confirming that apt completed successfully, but also by checking that the entire workflow is functioning correctly.

What to Check Before Going to Production

Before considering GitLab ready for ongoing production use, review these three short checklists.

First, check GitLab itself and access to it:

Once the core platform is secure, proceed to the Runner and CI checks:

The final checklist covers post-launch operations:

If all these checks pass, the infrastructure is ready not only to work today, but also to handle routine operational events such as updates, project growth, CI failures, and restoration from a backup.

This completes the technical part of the deployment. The remaining step is to summarize the results and determine which projects genuinely benefit from a self-hosted GitLab instance with their own Runner.

Conclusion

Self-hosted GitLab with its own Runner is well suited to teams and individual projects that require full control over repositories, CI/CD, and infrastructure. This setup is particularly appropriate for internal development, private enterprise projects, small teams, and situations where code or CI secrets should not be entrusted to an external service.

Initially, GitLab and the Runner can be hosted on the same VPS if pipelines run at a moderate frequency, only one job runs at a time, and CI does not create a heavy load. As the project grows, however, the Runner becomes a logical candidate for a separate server. This prevents builds from competing with GitLab for CPU, RAM, and disk I/O, while providing better isolation between code executed by CI and the primary instance.

The key is not to consider the setup complete after the first successful pipeline. For production environments, it is equally important to have backups with a tested restore process, monitor available disk space, secure CI/CD variables, apply updates, and establish clear Runner diagnostics. If these processes are set up in advance, self-hosted GitLab remains relatively straightforward to manage even after the initial test project evolves into a production system.

Thank you for your attention!

FAQ

Can GitLab and GitLab Runner run on the same VPS?

Yes. This is a perfectly viable setup for a small GitLab instance, a personal project, or a small team.

The key consideration is that during CI jobs, the Runner uses the same CPU, RAM, and disk resources as GitLab. You should therefore limit concurrency and container resource usage, then monitor the actual load.

If pipelines become resource-intensive or need to run in parallel, it is better to move the Runner to a separate VPS. For production GitLab instances, GitLab also recommends treating the Runner as separate infrastructure for running CI jobs.

Can multiple runners be connected to a single GitLab instance?

Yes. In fact, this is one of the primary ways to scale CI.

You can use project, group, and instance runners, with tags to route different types of jobs to each one. For example, one runner can handle standard Docker builds, another deployments, and a third specialized tasks.

What happens to a pipeline if the Runner is offline?

GitLab can create the pipeline and job, but no suitable Runner will pick up the job.

The job will typically remain pending until a suitable Runner becomes available again. If the Runner connects to GitLab after its service starts and all assignment conditions are still met, the job can proceed through its normal lifecycle.

Does a Runner survive restoring GitLab from a backup?

It can, but this depends on exactly what was preserved.

The Runner stores its local configuration separately, usually in:

   /etc/gitlab-runner/config.toml

If the Runner itself was not lost, its configuration and authentication data were preserved, and the corresponding instance data was restored in GitLab, the Runner may continue working without being re-registered.

After the restore, it is best to run the following command first:

   sudo gitlab-runner verify

and re-register the Runner only if there are issues.

Can I move a Runner to another server or connect it to a different GitLab instance?

Yes, but it is important to distinguish between moving the machine and switching to a different GitLab instance.

On a new server, Runner is installed from scratch and linked to the required configuration. GitLab allows multiple Runner registrations and the reuse of a suitable runner configuration. 

If the Runner needs to work with a different GitLab instance, you must configure it with that instance’s URL and complete the appropriate registration process. You cannot simply copy the existing config.toml file and expect its authentication credentials to work automatically with another GitLab instance.

What happens to a running job when the VPS is rebooted?

The job will be interrupted.

If GitLab Runner and Docker are running on the same server, a reboot will stop both the Runner service and any active containers. After the server boots, systemd may restart the Runner, but a job that was already running usually cannot simply resume from the same point.

In GitLab, the job will ultimately need to be rerun. Therefore, pipelines should be designed so that individual jobs are reproducible and do not depend on state unique to the previous container.

Does GitLab Runner require Docker?

No.

GitLab Runner supports multiple executors. Docker is just one option. For example, you can use the Shell executor, which runs commands directly on the host system.

This article uses the Docker executor because it provides a more predictable environment and better isolation of the CI job from the primary VPS. However, GitLab Runner itself does not require Docker to operate. 

Can GitLab be backed up while users are active?

Yes. GitLab’s built-in backup can be created while the instance is running, so routine backups do not require shutting down the entire instance each time.

However, a backup captures the state of the data at a specific point in time, while an active GitLab instance continues to change. For critical procedures, migrations, or particularly strict consistency requirements, plan a maintenance window and recovery strategy in advance.

Most importantly, test the restore itself. GitLab explicitly recommends testing the complete recovery process before it is needed in production. 

Sources

1. GitLab Docs — Install GitLab using the Linux package
2. GitLab Docs — Registering runners
3. GitLab Docs — Tutorial: Create, register, and run your own project runner
4. GitLab Docs — Back up and restore / Restore GitLab
5. GitLab Docs — GitLab Runner commands / Get started with GitLab Runner