Annotations

An annotation records something that explains your data: a deploy, a change, or a person's comment. When latency jumps at 14:02, an annotation at 14:01 saying checkout 1.42.0 answers the first question. CI, people and agents post annotations, and time charts draw them.

Marking a deploy is one call, after CI pushes the image:

curl -X POST https://telemetrymachine.com/api/organizations/<your-org>/annotations \
  -H "X-API-Key: $TM_INGEST_KEY" -H 'Content-Type: application/json' \
  -d '{
    "type": "deploy",
    "message": "checkout 1.42.0",
    "key": "[email protected]@production",
    "attributes": {
      "service.name": "checkout",
      "deployment.environment.name": "production"
    }
  }'

<your-org> is your organization's slug, the part after telemetrymachine.com/ when you're signed in. Every field, endpoint and error is in the API reference.

An annotation is a record, not an action. Posting one pages nobody and starts nothing. The system that owns what happened, such as your CI, stays the source of truth, and the annotation links to it.

Types

type says why the data might look different:

Earlier versions also had incident_begin, incident_end and other. They are gone: annotations posted with them are now comments, and a post with one gets a 400 that names the types.

Mark deploys from CI

CI can post with an ingest key, or with no stored secret at all (refer to Post from CI without a secret). A person's token doesn't belong in CI: it usually has wide scopes, and it stops working when that person leaves.

Ingest keys can't post annotations until you allow it, because they often end up inside apps and collectors. Give CI a key of its own, and turn annotate on for it:

tm-cli api-keys update apk_ID --annotate true

Store the key as a repository secret, TM_INGEST_KEY, and add a step after the deploy. It only warns if the post fails, so it never fails a deploy:

- name: Annotate the deploy in Telemetry Machine
  env:
    TM_INGEST_KEY: ${{ secrets.TM_INGEST_KEY }}
    RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
  run: |
    jq -n --arg sha "$GITHUB_SHA" --arg run "$RUN_URL" \
      '{type: "deploy", message: "checkout \($sha[0:7])", key: "checkout@\($sha)@production",
        attributes: {"service.name": "checkout", "deployment.environment.name": "production",
                     "vcs.ref.head.revision": $sha, "url.full": $run}}' |
    curl --fail-with-body -sS --retry 3 -X POST \
      https://telemetrymachine.com/api/organizations/<your-org>/annotations \
      -H "X-API-Key: $TM_INGEST_KEY" -H 'Content-Type: application/json' -d @- ||
      echo "::warning::Couldn't annotate the deploy in Telemetry Machine"

The key makes a retry safe: posting it again returns the first annotation (refer to Keys). If a leaked key posts markers, each annotation records the key that posted it, so you can find and delete them, and revoke the key.

What to send with a deploy

A deploy needs service.name or vcs.ref.head.revision. Send the values that your service's own telemetry carries, so the deploy lines up with it:

The API reference lists the rules that each one must meet.

Post from CI without a secret

GitHub Actions and GitLab CI give each job a short-lived OIDC token, signed by the provider, that says which repository, branch and environment the job runs for. Telemetry Machine accepts that token in place of an ingest key, for the repositories that you trust:

• Nothing is stored in CI, so nothing can leak from it. A token that escapes into a log works for a few minutes, only against Telemetry Machine, and only for that repository.
• The repository and the branch are proven, not claimed. A pull request from a fork gets no token, and a rule on refs/heads/main refuses other branches.
• Each annotation names the run that posted it, in created_by_ci.

This works with github.com and gitlab.com. GitHub Enterprise Server and self-managed GitLab aren't supported yet.

1. Trust the repository

An owner or admin adds a trust rule for the repository, with a personal access token. There's no settings page for rules yet.

curl -X POST https://telemetrymachine.com/api/organizations/<your-org>/ci-trust-rules \
  -H "Authorization: Bearer $TM_TOKEN" -H 'Content-Type: application/json' \
  -d '{ "provider": "github", "repository": "acme/checkout", "ref": "refs/heads/main" }'

A rule can also name an environment, and a job must match everything that the rule names. To let in the jobs on main and also the jobs in production, create two rules. A job whose ref changes with each release, such as one that runs on a release's tag, is best matched by its environment. The API reference has every field.

2. Send the job's token

The job asks its provider for a token addressed to telemetrymachine.com, and sends it as Authorization: Bearer in place of X-API-Key:

permissions:
  contents: read
  id-token: write
steps:
  - name: Annotate the deploy in Telemetry Machine
    env:
      RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
    run: |
      TOKEN=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
        "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=telemetrymachine.com" | jq -r .value)
      jq -n --arg sha "$GITHUB_SHA" --arg run "$RUN_URL" \
        '{type: "deploy", message: "checkout \($sha[0:7])", key: "checkout@\($sha)@production",
          attributes: {"service.name": "checkout", "deployment.environment.name": "production",
                       "vcs.ref.head.revision": $sha, "url.full": $run}}' |
      curl --fail-with-body -sS --retry 3 -X POST \
        https://telemetrymachine.com/api/organizations/<your-org>/annotations \
        -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d @- ||
        echo "::warning::Couldn't annotate the deploy in Telemetry Machine"

annotate-deploy:
  id_tokens:
    TM_ID_TOKEN:
      aud: telemetrymachine.com
  script:
    - |
      curl --fail-with-body -sS --retry 3 -X POST \
        "https://telemetrymachine.com/api/organizations/<your-org>/annotations" \
        -H "Authorization: Bearer $TM_ID_TOKEN" -H 'Content-Type: application/json' \
        --data @- <<EOF || echo "Couldn't annotate the deploy in Telemetry Machine"
      {
        "type": "deploy",
        "message": "checkout $CI_COMMIT_SHORT_SHA",
        "key": "checkout@$CI_COMMIT_SHA@production",
        "attributes": {
          "service.name": "checkout",
          "deployment.environment.name": "production",
          "vcs.ref.head.revision": "$CI_COMMIT_SHA",
          "url.full": "$CI_PIPELINE_URL"
        }
      }
      EOF

A token that isn't valid gets 401, and a job that no rule lets in gets 403. A job's token may create annotations and retry its own keys, but not update or delete them: to end a window later, use an ingest key or a person's token.

Mark a window

A load test or an outage covers a stretch of time. Send ended_at with the annotation, or set it when the window ends:

curl -X PATCH https://telemetrymachine.com/api/organizations/<your-org>/annotations/ann_ID \
  -H "Authorization: Bearer $TM_TOKEN" -H 'Content-Type: application/json' \
  -d '{ "ended_at": "2026-10-26T15:00:00Z" }'

Without ended_at, an annotation is a moment. Nothing ends one by itself.

On charts

Time charts on dashboards and widgets mark the annotations of their dataset and the organization's with a line at their start; hover it to read them. A chart filtered to one value of a field, such as k8s.node.name, leaves out annotations naming another value of it.

Name attributes after your data

Use attribute names that already appear in your telemetry, so an annotation lines up with the services, pods and hosts it affected: service.name, service.version, deployment.environment.name, k8s.deployment.name, k8s.pod.name, host.name. Check a name against the OpenTelemetry semantic conventions (or tm-cli semconv search) rather than inventing one. Apart from a deploy's, Telemetry Machine doesn't check the names, so a typo just won't match anything.

From tm-cli and agents

tm-cli annotations create --type change --message "raised the DB pool to 50" \
  --attr service.name=checkout --time 10m
tm-cli annotations list --since 7d --type deploy
tm-cli annotations update ann_ID --org-wide
tm-cli annotations delete ann_ID

Agents connected to the Telemetry Machine MCP server (refer to Set up with a coding agent) post with the post_annotation tool and read with get_annotations. An agent working on your systems should post a change or deploy annotation for each change it makes, so the next person or agent looking at the data can see it.

People and agents need a token with the annotations:write scope ("Post and edit annotations") to post, edit or delete; anyone in the organization can list them.