How to use GitLab Development Kit
GitLab Observability Backend

GitLab Observability Backend

Configure the GitLab Observability Backend (GOB) to run locally in GDK. This is required to ingest and query observability signals metrics, logs, and traces. Using GOB in GDK will help you get up to speed quickly when developing these features.

Learn more about Observability for tracing, metrics and logs in our documentation.

Prerequisites

  • ClickHouse enabled in your GDK. [ Docs]
  • An EE Ultimate license in your GDK.
  • GDK running as SaaS. [ Docs]

Get started

  1. Enable the observability feature flag. Feature.enable(:observability_features)

  2. Enable GOB in GDK by updating gdk.yml by running the following command. It is not enabled by default.

       gdk config set gitlab_observability_backend.enabled true

  3. Set the required environment variables by adding the following lines to <gdk_root>/env.runit

    export OVERRIDE_OBSERVABILITY_QUERY_URL=http://localhost:9003
export OVERRIDE_OBSERVABILITY_INGEST_URL=http://localhost:4318

  4. Run gdk reconfigure.

  5. Run gdk start to start the new service.

Troubleshooting

gitlab-observability-backend service not starting on macOS

After following the steps above you might find that gitlab-observability-backend is not able to start and returns the following message:

gdk status gitlab-observability-backend

down: /gitlab-development-kit/services/gitlab-observability-backend: 4s; run: log: (pid 93406) 5494s

In this case, check if the process can start at all by launching it manually:

cd gitlab/gitlab-observability-backend/go/cmd/all-in-one 
./all-in-one

./all-in-one [1] 19695 killed # the process is killed

If the process is killed right after starting, it might be due to issues with macOS code signing. To check if this is the case, run codesign:

codesign -vv all-in-one
all-in-one: invalid signature (code or signature have been modified)
In architecture: arm64

If an invalid signature is found, the process won’t be able to start.

Though it is not entirely clear why this happens (a proper fix TBD), in some cases it was because go was being managed by asdf. See this issue where other users faced a similar problem.

A workaround for this issue is to explicitly install or use a different go installation (for example via homebrew or go.dev) and build the service with it.

brew install go

/opt/homebrew/bin/go build .

codesign -vv all-in-one
all-in-one: valid on disk
all-in-one: satisfies its Designated Requirement

./all-in-one

Error: get connection: getting database handle ... # process actually gets started now

At this point it should be possible to run gdk start gitlab-observability-backend successfully.

Note that every time you run gdk update or gdk reconfigure, you must manually rebuild the gitlab-observability-backend.

Last updated on
GitLab Geo - Advanced InstallationGitLab version