Instrument Go net/http with OpenTelemetry

Send traces, logs and metrics from a Go net/http server to Telemetry Machine with otelhttp, slog and the OpenTelemetry Go SDK.

Open in Claude

otelhttp traces every request and records HTTP metrics, and the otelslog bridge sends your slog records as logs. Go has no zero-code instrumentation, so you add one setup file and wrap your handler.

Create the datasets

Each signal goes to its own dataset. Create three, and an ingest key that can write to all of them, with tm-cli or the MCP server:

$ tm-cli datasets create --name shop-prod-traces --type traces
$ tm-cli datasets create --name shop-prod-logs --type logs
$ tm-cli datasets create --name shop-prod-metrics --type metrics
$ tm-cli api-keys create --name shop-api --dataset shop-prod-traces --dataset shop-prod-logs --dataset shop-prod-metrics

tm-cli shows the key once, in api_key.api_key. Over MCP, your agent gets it once and saves it in your app's configuration. In the app, create each dataset, then a key for all three under API keys in your organization's settings.

Install

Add otel.go to your main package. It sets up an exporter for each signal from the OTEL_* environment variables:

otel.go
package main

import (
	"context"
	"errors"
	"net/http"
	"strings"

	"go.opentelemetry.io/contrib/exporters/autoexport"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/log/global"
	"go.opentelemetry.io/otel/propagation"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

// setupOTel starts sending traces, metrics and logs. Call the function it
// returns before the program exits, to send what is still buffered.
func setupOTel(ctx context.Context) (func(context.Context) error, error) {
	spans, err := autoexport.NewSpanExporter(ctx)
	if err != nil {
		return nil, err
	}
	metrics, err := autoexport.NewMetricReader(ctx)
	if err != nil {
		return nil, err
	}
	logs, err := autoexport.NewLogExporter(ctx)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(spans))
	mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(metrics))
	lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(sdklog.NewBatchProcessor(logs)))
	otel.SetTracerProvider(tp)
	otel.SetMeterProvider(mp)
	global.SetLoggerProvider(lp)
	otel.SetTextMapPropagator(propagation.TraceContext{})
	return func(ctx context.Context) error {
		return errors.Join(tp.Shutdown(ctx), mp.Shutdown(ctx), lp.Shutdown(ctx))
	}, nil
}

// withRoute sets http.route on the request's span, from the pattern that
// http.ServeMux matched.
func withRoute(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		next.ServeHTTP(w, r)
		if i := strings.IndexByte(r.Pattern, '/'); i >= 0 {
			trace.SpanFromContext(r.Context()).SetAttributes(attribute.String("http.route", r.Pattern[i:]))
		}
	})
}

In main.go, call setupOTel, wrap your mux, log with an otelslog logger, and shut down cleanly on SIGTERM:

main.go
package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"

	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

var logger = otelslog.NewLogger("shop")

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	shutdown, err := setupOTel(ctx)
	if err != nil {
		log.Fatal(err)
	}
	defer shutdown(context.Background())

	mux := http.NewServeMux()
	mux.HandleFunc("GET /items/{id}", func(w http.ResponseWriter, r *http.Request) {
		logger.InfoContext(r.Context(), "reading item", "id", r.PathValue("id"))
		w.Write([]byte(r.PathValue("id")))
	})

	srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(withRoute(mux), "server")}
	go func() {
		<-ctx.Done()
		srv.Shutdown(context.Background())
	}()
	if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
		shutdown(context.Background())
		log.Fatal(err)
	}
}

Then fetch the modules:

Terminal
$ go mod tidy

Configure

Shell
export TM_INGEST_KEY='<ingest-key>'
export OTEL_SERVICE_NAME=shop-api
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.telemetrymachine.com
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="X-API-Key=$TM_INGEST_KEY,X-Dataset-Name=shop-prod-traces"
export OTEL_EXPORTER_OTLP_LOGS_HEADERS="X-API-Key=$TM_INGEST_KEY,X-Dataset-Name=shop-prod-logs"
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="X-API-Key=$TM_INGEST_KEY,X-Dataset-Name=shop-prod-metrics"

Run

Terminal
$ go run .

Check it arrived

Send your app a few requests, then open each explorer and choose the dataset:

  • Traces, shop-prod-traces: one server span per request, named after its pattern, such as GET /items/{id}.
  • Logs, shop-prod-logs: your slog records, each with its request's trace.trace_id.
  • Metrics, shop-prod-metrics: http.server.request.duration, sent once a minute.

Or count spans by name with tm-cli or the MCP server:

$ tm-cli query shop-prod-traces --since 15m --select "name, count(*)" --group-by name

Troubleshooting

  • Spans have no http.route: wrap the mux in withRoute inside otelhttp.NewHandler, as above. otelhttp reads the route before ServeMux has matched one.
  • The last spans and logs before exit don't arrive: the deferred shutdown didn't run. log.Fatal and os.Exit skip deferred calls, so return from main or call shutdown first, as above.
  • Logs have no trace.trace_id: pass the request's context, as in logger.InfoContext(r.Context(), ...).