Dagger pipelines and OpenTelemetry environment

Recently I tried to generate tracing data out of the pipeline for zerokspot.com and feed that into an external collector. Doing that with the ci/main.go itself was easy: Just set the relevant environment variables, integrate the OpenTelemetry SDK and off we go.

But now I also wanted to forward the trace ID and collector configuration into the blogbinary so that it could also generate traces and send those to the same collector. That binary is first built by the pipeline and later used for things like generating yearly archives and the blogroll for the site.

There I noticed something unexpected: Turns out that BuildKit, which is used by Dagger, injects its own OTEL_* environment variables into every container execution.


… will produce something like that:

#4 env

Setting my own OTEL_EXPORTER_OLTP_TRACES_{ENDPOINT,HEADERS,PROTOCOL} variables won’t work. BuildKit will always override them. This was also confirmed on Discord. For now, I’ve solved this by just injecting some custom environment variables and then mapping them inside the blog binary to what’s expected by the OpenTelemetry SDK.

Dagger/BuildKit comes with a built-in OTEL collector but I cannot use that in my current setup, so I will most likely stick with my little workaround for now. That being said, I will definitely keep an eye on future changes to that integration!

You can also find this post linked to from the following pages: