[Go] Genkit overrides global OTel TracerProvider after Init/flow, breaking Pyroscope instrumentation #3709
Description
Describe the bug
Genkit overrides the global OpenTelemetry tracer provider when the host application uses Pyroscope's otel-profiling-go wrapper provider. After calling genkit.Init and running a flow, otel.GetTracerProvider() changes from the Pyroscope wrapper to a plain SDK provider, which disables Pyroscope instrumentation.
Observed output from the minimal repro below:
Before genkit: *otelpyroscope.tracerProvider (0x40001910e0)
After genkit: *trace.TracerProvider (0x4000146630)
This indicates Genkit replaced the global provider that had been set to the Pyroscope wrapper.
To Reproduce
- Use the following minimal program (Go) which sets up an SDK tracer provider, wraps it with Pyroscope's provider, then runs a genkit flow and compares before/after types of
otel.GetTracerProvider().
package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"time"
"github.com/firebase/genkit/go/genkit"
otelpyroscope "github.com/grafana/otel-profiling-go"
"github.com/grafana/pyroscope-go"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/stdout/stdouttrace"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/trace"
)
func main() {
ctx := context.Background()
tracerProvider, shutdown, err := initTracer(ctx)
if err != nil {
log.Fatalf("Failed to initialize tracer: %v", err)
}
defer shutdown(ctx)
// Wrap SDK provider with Pyroscope provider
otel.SetTracerProvider(otelpyroscope.NewTracerProvider(tracerProvider))
_, err = pyroscope.Start(pyroscope.Config{
ApplicationName: "my-service",
ServerAddress: "http://localhost:4040",
})
if err != nil {
log.Fatalf("Failed to initialize pyroscope: %v", err)
}
fmt.Printf("Before genkit: %T (%p)\n", otel.GetTracerProvider(), otel.GetTracerProvider())
gkit := genkit.Init(ctx)
testFlow := genkit.DefineFlow(gkit, "test", func(ctx context.Context, input string) (string, error) {
return input, nil
})
testFlow.Run(ctx, "foo")
fmt.Printf("After genkit: %T (%p)\n", otel.GetTracerProvider(), otel.GetTracerProvider())
}
func initTracer(ctx context.Context) (trace.TracerProvider, func(context.Context) error, error) {
var shutdownFuncs []func(context.Context) error
var err error
shutdown := func(ctx context.Context) error {
var err error
for _, fn := range shutdownFuncs {
err = errors.Join(err, fn(ctx))
}
shutdownFuncs = nil
return err
}
handleErr := func(inErr error) {
err = errors.Join(inErr, shutdown(ctx))
}
tracerProvider, err := newTracerProvider()
if err != nil {
handleErr(err)
return tracerProvider, shutdown, err
}
shutdownFuncs = append(shutdownFuncs, tracerProvider.Shutdown)
otel.SetTracerProvider(tracerProvider)
return tracerProvider, shutdown, err
}
func newTracerProvider() (*sdktrace.TracerProvider, error) {
traceExporter, err := stdouttrace.New(stdouttrace.WithWriter(io.Discard))
if err != nil {
return nil, err
}
tracerProvider := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(traceExporter, sdktrace.WithBatchTimeout(time.Second)),
)
return tracerProvider, nil
}- Run:
export GENKIT_ENV=dev && genkit ui:start && genkit start -- go run main.go- Observe that the global provider type has changed away from the Pyroscope wrapper.
Expected behavior
- Genkit should not override a pre-configured global tracer provider.
- At minimum, if the current provider is not an SDK provider, Genkit should avoid replacing it and document how to integrate exporters.
- Genkit UI should work even if the host application has its own OpenTelemetry setup.
Screenshots
N/A (see console output above).
Runtime (please complete the following information):
- OS: macOS
- Version: 15.7.1
Go version
go version go1.25.2 darwin/arm64
Additional context
-
Module versions:
github.com/firebase/genkit/go v1.0.5github.com/grafana/otel-profiling-go v0.5.1github.com/grafana/pyroscope-go v1.2.7go.opentelemetry.io/otel v1.38.0go.opentelemetry.io/otel/sdk v1.38.0go.opentelemetry.io/otel/trace v1.38.0go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.38.0
-
Likely cause in Genkit (v1.0.5):
core/tracing/tracing.gounconditionally sets a new SDK provider if the current provider is not exactly*sdktrace.TracerProvider, then returns it by asserting the global provider back to*sdktrace.TracerProvider:
func TracerProvider() *sdktrace.TracerProvider {
if tp := otel.GetTracerProvider(); tp != nil {
if sdkTP, ok := tp.(*sdktrace.TracerProvider); ok {
return sdkTP
}
}
providerInitOnce.Do(func() {
otel.SetTracerProvider(sdktrace.NewTracerProvider())
if telemetryURL := os.Getenv("GENKIT_TELEMETRY_SERVER"); telemetryURL != "" {
WriteTelemetryImmediate(NewHTTPTelemetryClient(telemetryURL))
}
})
return otel.GetTracerProvider().(*sdktrace.TracerProvider)
}- This replaces wrapper providers such as
otelpyroscope.tracerProvider, and the final type assertion can panic if a non-SDK provider is still installed.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status