Send OpenTelemetry Data

Telemetry Machine receives standard OTLP, so the OpenTelemetry docs apply as written. This page lists only what is particular to Telemetry Machine.

Endpoint and protocol

Telemetry Machine rejects OTLP JSON and does not expose OTLP over gRPC. It accepts gzip-compressed bodies.

/v1/events takes JSON events, not OTLP. See Send JSON Events.

One dataset per signal

Each dataset holds one type of data: traces, logs, metrics, or events. OTLP goes to traces, logs, and metrics datasets; events datasets take JSON instead (see Send JSON Events). Telemetry Machine routes each request by its X-Dataset-Name header, so every signal needs its own header. Set the per-signal variables, not OTEL_EXPORTER_OTLP_HEADERS:

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.telemetrymachine.com
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_HEADERS=X-API-Key=<ingest-key>,X-Dataset-Name=shop-prod-traces
OTEL_EXPORTER_OTLP_LOGS_HEADERS=X-API-Key=<ingest-key>,X-Dataset-Name=shop-prod-logs

Name datasets <app>-<environment>-<signal>, such as shop-prod-traces. All the services of one app share its datasets; service.name tells them apart.

Collector

A Collector needs one otlp_http exporter for each dataset, and a pipeline for each signal that uses that signal's exporter. Collector versions before v0.144.0 call this exporter otlphttp:

exporters:
  otlp_http/traces:
    endpoint: https://api.telemetrymachine.com
    headers:
      X-API-Key: ${env:TM_INGEST_KEY}
      X-Dataset-Name: shop-prod-traces
  otlp_http/logs:
    endpoint: https://api.telemetrymachine.com
    headers:
      X-API-Key: ${env:TM_INGEST_KEY}
      X-Dataset-Name: shop-prod-logs

service:
  pipelines:
    traces:
      exporters: [otlp_http/traces]
    logs:
      exporters: [otlp_http/logs]

AWS Lambda

Follow the OpenTelemetry docs for the Lambda layer. By default, the layer's Collector does not export to Telemetry Machine. Give it a Collector config with the exporters above, and set OPENTELEMETRY_COLLECTOR_CONFIG_URI to that file. Leave OTEL_EXPORTER_OTLP_ENDPOINT unset, so the function sends to the layer's Collector, which exports after the function returns.

Attribute and metric names

Before you name an attribute or a metric, look it up in the OpenTelemetry semantic conventions, and use the standard name, type and values when one fits. Agents can search them with the Telemetry Machine MCP server's semconv tool or with tm-cli semconv search "<words>". Put an attribute that has no standard name under your app's own namespace, such as app.checkout.cart_size.

Exceptions and span events

Telemetry Machine keeps a span's events in one JSON column, events, which queries cannot group by. Record exceptions as log records instead of span events: severity ERROR with the exception.* attributes, emitted while the span is active so the log carries its trace ID. Also set the span's status to error.

Error slugs

Give each place that raises or handles an error a fixed ID, set as the exception.slug attribute on the span and on the exception's log record, such as err-payment-declined. You can search the code for a slug, group errors by it, and find errors that nobody handled:

"status_code" = 2 AND "attr.exception.slug" IS NULL

Prometheus metrics

Telemetry Machine has no Prometheus remote_write endpoint. Scrape the targets you need with the Collector's prometheus receiver and send them over OTLP to a metrics dataset.

Rust

Enable the reqwest-rustls feature of opentelemetry-otlp. Without a TLS backend, export to https://api.telemetrymachine.com fails, and the SDK reports that only through tracing::error!. Keep a tracing_subscriber::fmt layer during setup, so you see export errors.

How fields are stored

Attributes become columns: attr.* for span, log, and data-point attributes, and resource.* for resource attributes. Their values are stored as strings. See Schema Management.

OpenTelemetry docs