# Instrument Go net/http with OpenTelemetry

> Send traces, logs and metrics from a Go net/http server to Telemetry Machine with otelhttp, slog and the OpenTelemetry Go SDK.

`otelhttp` traces every request and records HTTP metrics, and the `otelslog` bridge sends your `slog` records as logs. Go has no zero-code instrumentation, so you add one setup file and wrap your handler.

## Create the datasets

Each signal goes to its own dataset. Create three, and an ingest key that can write to all of them, with [tm-cli](https://telemetrymachine.com/docs/quick-start#tm-cli) or the [MCP server](https://telemetrymachine.com/docs/quick-start#option-1-set-up-with-a-coding-agent):

```console tab="tm-cli"
$ tm-cli datasets create --name shop-prod-traces --type traces
$ tm-cli datasets create --name shop-prod-logs --type logs
$ tm-cli datasets create --name shop-prod-metrics --type metrics
$ tm-cli api-keys create --name shop-api --dataset shop-prod-traces --dataset shop-prod-logs --dataset shop-prod-metrics
```

```json tab="MCP"
create_dataset { "name": "shop-prod-traces", "signal_type": "traces" }
create_dataset { "name": "shop-prod-logs", "signal_type": "logs" }
create_dataset { "name": "shop-prod-metrics", "signal_type": "metrics" }
create_ingest_key {
  "name": "shop-api",
  "datasets": ["shop-prod-traces", "shop-prod-logs", "shop-prod-metrics"]
}
```

tm-cli shows the key once, in `api_key.api_key`. Over MCP, your agent gets it once and saves it in your app's configuration. In the app, [create each dataset](https://telemetrymachine.com/datasets/new), then a key for all three under **API keys** in your organization's settings.

## Install

Add `otel.go` to your main package. It sets up an exporter for each signal from the `OTEL_*` environment variables:

```go title="otel.go"
package main

import (
	"context"
	"errors"
	"net/http"
	"strings"

	"go.opentelemetry.io/contrib/exporters/autoexport"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/log/global"
	"go.opentelemetry.io/otel/propagation"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

// setupOTel starts sending traces, metrics and logs. Call the function it
// returns before the program exits, to send what is still buffered.
func setupOTel(ctx context.Context) (func(context.Context) error, error) {
	spans, err := autoexport.NewSpanExporter(ctx)
	if err != nil {
		return nil, err
	}
	metrics, err := autoexport.NewMetricReader(ctx)
	if err != nil {
		return nil, err
	}
	logs, err := autoexport.NewLogExporter(ctx)
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(spans))
	mp := sdkmetric.NewMeterProvider(sdkmetric.WithReader(metrics))
	lp := sdklog.NewLoggerProvider(sdklog.WithProcessor(sdklog.NewBatchProcessor(logs)))
	otel.SetTracerProvider(tp)
	otel.SetMeterProvider(mp)
	global.SetLoggerProvider(lp)
	otel.SetTextMapPropagator(propagation.TraceContext{})
	return func(ctx context.Context) error {
		return errors.Join(tp.Shutdown(ctx), mp.Shutdown(ctx), lp.Shutdown(ctx))
	}, nil
}

// withRoute sets http.route on the request's span, from the pattern that
// http.ServeMux matched.
func withRoute(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		next.ServeHTTP(w, r)
		if i := strings.IndexByte(r.Pattern, '/'); i >= 0 {
			trace.SpanFromContext(r.Context()).SetAttributes(attribute.String("http.route", r.Pattern[i:]))
		}
	})
}
```

In `main.go`, call `setupOTel`, wrap your mux, log with an `otelslog` logger, and shut down cleanly on `SIGTERM`:

```go title="main.go" {16,19-25,29,33-41}
package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"

	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

var logger = otelslog.NewLogger("shop")

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	shutdown, err := setupOTel(ctx)
	if err != nil {
		log.Fatal(err)
	}
	defer shutdown(context.Background())

	mux := http.NewServeMux()
	mux.HandleFunc("GET /items/{id}", func(w http.ResponseWriter, r *http.Request) {
		logger.InfoContext(r.Context(), "reading item", "id", r.PathValue("id"))
		w.Write([]byte(r.PathValue("id")))
	})

	srv := &http.Server{Addr: ":8080", Handler: otelhttp.NewHandler(withRoute(mux), "server")}
	go func() {
		<-ctx.Done()
		srv.Shutdown(context.Background())
	}()
	if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
		shutdown(context.Background())
		log.Fatal(err)
	}
}
```

Then fetch the modules:

```console
$ go mod tidy
```

## Configure

```bash
export TM_INGEST_KEY='<ingest-key>'
export OTEL_SERVICE_NAME=shop-api
export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.telemetrymachine.com
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="X-API-Key=$TM_INGEST_KEY,X-Dataset-Name=shop-prod-traces"
export OTEL_EXPORTER_OTLP_LOGS_HEADERS="X-API-Key=$TM_INGEST_KEY,X-Dataset-Name=shop-prod-logs"
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="X-API-Key=$TM_INGEST_KEY,X-Dataset-Name=shop-prod-metrics"
```

## Run

```console
$ go run .
```

## Check it arrived

Send your app a few requests, then open each explorer and choose the dataset:

- **Traces**, `shop-prod-traces`: one server span per request, named after its pattern, such as `GET /items/{id}`.
- **Logs**, `shop-prod-logs`: your `slog` records, each with its request's `trace.trace_id`.
- **Metrics**, `shop-prod-metrics`: `http.server.request.duration`, sent once a minute.

Or count spans by name with tm-cli or the MCP server:

```console tab="tm-cli"
$ tm-cli query shop-prod-traces --since 15m --select "name, count(*)" --group-by name
```

```json tab="MCP"
run_query {
  "dataset": "shop-prod-traces",
  "description": "Spans by name",
  "since": "15m",
  "select": "name, count(*)",
  "group_by": "name"
}
```

## Troubleshooting

- **Spans have no `http.route`:** wrap the mux in `withRoute` inside `otelhttp.NewHandler`, as above. `otelhttp` reads the route before `ServeMux` has matched one.
- **The last spans and logs before exit don't arrive:** the deferred `shutdown` didn't run. `log.Fatal` and `os.Exit` skip deferred calls, so return from `main` or call `shutdown` first, as above.
- **Logs have no `trace.trace_id`:** pass the request's context, as in `logger.InfoContext(r.Context(), ...)`.

---

All pages: https://telemetrymachine.com/docs/llms.txt