# Annotations > Mark deploys, changes and comments next to your telemetry, from CI, tm-cli or an agent. An annotation records something that explains your data: a deploy, a change, or a person's comment. When latency jumps at 14:02, an annotation at 14:01 saying `checkout 1.42.0` answers the first question. CI, people and agents post annotations, and time charts draw them. Marking a deploy is one call, after CI pushes the image: ```sh curl -X POST https://telemetrymachine.com/api/organizations//annotations \ -H "X-API-Key: $TM_INGEST_KEY" -H 'Content-Type: application/json' \ -d '{ "type": "deploy", "message": "checkout 1.42.0", "key": "checkout@1.42.0@production", "attributes": { "service.name": "checkout", "deployment.environment.name": "production" } }' ``` `` is your organization's slug, the part after `telemetrymachine.com/` when you're signed in. Every field, endpoint and error is in the [API reference](https://telemetrymachine.com/docs/api/annotations). An annotation is a record, not an action. Posting one pages nobody and starts nothing. The system that owns what happened, such as your CI, stays the source of truth, and the annotation links to it. ## Types `type` says why the data might look different: | `type` | Use it for | For example | | --------------- | --------------------------------------- | ------------------------------------------------ | | _deploy_ | New code started to run | an image push, a rollback | | _change_ | A deliberate change that isn't new code | a config edit, a feature flag, a manual scale-up | | _comment_ | A person's note on the timeline | "this spike was the load test" | Earlier versions also had `incident_begin`, `incident_end` and `other`. They are gone: annotations posted with them are now comments, and a post with one gets a 400 that names the types. ## Mark deploys from CI CI can post with an [ingest key](https://telemetrymachine.com/docs/send-opentelemetry), or with no stored secret at all (refer to [Post from CI without a secret](https://telemetrymachine.com/docs/annotations#post-from-ci-without-a-secret)). A person's token doesn't belong in CI: it usually has wide scopes, and it stops working when that person leaves. Ingest keys can't post annotations until you allow it, because they often end up inside apps and collectors. Give CI a key of its own, and turn annotate on for it: ```sh tm-cli api-keys update apk_ID --annotate true ``` Store the key as a repository secret, `TM_INGEST_KEY`, and add a step after the deploy. It only warns if the post fails, so it never fails a deploy: ```yaml title=".github/workflows/publish.yml" - name: Annotate the deploy in Telemetry Machine env: TM_INGEST_KEY: ${{ secrets.TM_INGEST_KEY }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | jq -n --arg sha "$GITHUB_SHA" --arg run "$RUN_URL" \ '{type: "deploy", message: "checkout \($sha[0:7])", key: "checkout@\($sha)@production", attributes: {"service.name": "checkout", "deployment.environment.name": "production", "vcs.ref.head.revision": $sha, "url.full": $run}}' | curl --fail-with-body -sS --retry 3 -X POST \ https://telemetrymachine.com/api/organizations//annotations \ -H "X-API-Key: $TM_INGEST_KEY" -H 'Content-Type: application/json' -d @- || echo "::warning::Couldn't annotate the deploy in Telemetry Machine" ``` The `key` makes a retry safe: posting it again returns the first annotation (refer to [Keys](https://telemetrymachine.com/docs/api/annotations#keys)). If a leaked key posts markers, each annotation records the key that posted it, so you can find and delete them, and revoke the key. ### What to send with a deploy A deploy needs `service.name` or `vcs.ref.head.revision`. Send the values that your service's own telemetry carries, so the deploy lines up with it: | Attribute | What it does | | ----------------------------- | -------------------------------------------------------------------------------------------- | | `service.name` | Puts the deploy on that service's charts. A chart filtered to another service leaves it out. | | `deployment.environment.name` | Keeps a staging deploy off production charts. The key ends with it for the same reason. | | `vcs.ref.head.revision` | The commit that it ships, so you and your agents can find the spans that ran it. | | `service.version` | The version that it ships, as the service reports it. | | `url.full` | A link to the CI run, for whoever wants the build log. | The [API reference](https://telemetrymachine.com/docs/api/annotations#deploy-attributes) lists the rules that each one must meet. ## Post from CI without a secret GitHub Actions and GitLab CI give each job a short-lived OIDC token, signed by the provider, that says which repository, branch and environment the job runs for. Telemetry Machine accepts that token in place of an ingest key, for the repositories that you trust: - **Nothing is stored in CI,** so nothing can leak from it. A token that escapes into a log works for a few minutes, only against Telemetry Machine, and only for that repository. - **The repository and the branch are proven, not claimed.** A pull request from a fork gets no token, and a rule on `refs/heads/main` refuses other branches. - **Each annotation names the run that posted it,** in `created_by_ci`. This works with github.com and gitlab.com. GitHub Enterprise Server and self-managed GitLab aren't supported yet. ### 1. Trust the repository An owner or admin adds a trust rule for the repository, with a personal access token. There's no settings page for rules yet. ```sh title="Trust the main branch of acme/checkout" curl -X POST https://telemetrymachine.com/api/organizations//ci-trust-rules \ -H "Authorization: Bearer $TM_TOKEN" -H 'Content-Type: application/json' \ -d '{ "provider": "github", "repository": "acme/checkout", "ref": "refs/heads/main" }' ``` A rule can also name an `environment`, and a job must match everything that the rule names. To let in the jobs on `main` and also the jobs in `production`, create two rules. A job whose ref changes with each release, such as one that runs on a release's tag, is best matched by its environment. The [API reference](https://telemetrymachine.com/docs/api/ci-trust-rules) has every field. ### 2. Send the job's token The job asks its provider for a token addressed to `telemetrymachine.com`, and sends it as `Authorization: Bearer` in place of `X-API-Key`: ```yaml tab="GitHub Actions" title=".github/workflows/publish.yml" permissions: contents: read id-token: write steps: - name: Annotate the deploy in Telemetry Machine env: RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} run: | TOKEN=$(curl -sS -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \ "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=telemetrymachine.com" | jq -r .value) jq -n --arg sha "$GITHUB_SHA" --arg run "$RUN_URL" \ '{type: "deploy", message: "checkout \($sha[0:7])", key: "checkout@\($sha)@production", attributes: {"service.name": "checkout", "deployment.environment.name": "production", "vcs.ref.head.revision": $sha, "url.full": $run}}' | curl --fail-with-body -sS --retry 3 -X POST \ https://telemetrymachine.com/api/organizations//annotations \ -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d @- || echo "::warning::Couldn't annotate the deploy in Telemetry Machine" ``` ```yaml tab="GitLab CI" title=".gitlab-ci.yml" annotate-deploy: id_tokens: TM_ID_TOKEN: aud: telemetrymachine.com script: - | curl --fail-with-body -sS --retry 3 -X POST \ "https://telemetrymachine.com/api/organizations//annotations" \ -H "Authorization: Bearer $TM_ID_TOKEN" -H 'Content-Type: application/json' \ --data @- </annotations/ann_ID \ -H "Authorization: Bearer $TM_TOKEN" -H 'Content-Type: application/json' \ -d '{ "ended_at": "2026-10-26T15:00:00Z" }' ``` Without `ended_at`, an annotation is a moment. Nothing ends one by itself. ## On charts Time charts on dashboards and widgets mark the annotations of their dataset and the organization's with a line at their start; hover it to read them. A chart filtered to one value of a field, such as `k8s.node.name`, leaves out annotations naming another value of it. ## Name attributes after your data Use attribute names that already appear in your telemetry, so an annotation lines up with the services, pods and hosts it affected: `service.name`, `service.version`, `deployment.environment.name`, `k8s.deployment.name`, `k8s.pod.name`, `host.name`. Check a name against the [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) (or `tm-cli semconv search`) rather than inventing one. Apart from a deploy's, Telemetry Machine doesn't check the names, so a typo just won't match anything. ## From tm-cli and agents ```sh tm-cli annotations create --type change --message "raised the DB pool to 50" \ --attr service.name=checkout --time 10m tm-cli annotations list --since 7d --type deploy tm-cli annotations update ann_ID --org-wide tm-cli annotations delete ann_ID ``` Agents connected to the Telemetry Machine MCP server (refer to [Set up with a coding agent](https://telemetrymachine.com/docs/quick-start#option-1-set-up-with-a-coding-agent)) post with the `post_annotation` tool and read with `get_annotations`. An agent working on your systems should post a `change` or `deploy` annotation for each change it makes, so the next person or agent looking at the data can see it. People and agents need a token with the **annotations:write** scope ("Post and edit annotations") to post, edit or delete; anyone in the organization can list them. --- All pages: https://telemetrymachine.com/docs/llms.txt