Effective Investigation
The order in which to look at Telemetry Machine data when something is wrong.
When something is wrong, look in this order. Writing Effective Queries covers the queries themselves.
- Check alerting first. What already fired tells you where to look.
- Check SLOs. An SLO that is burning its error budget names the service and the kind of failure.
- Find what the bad events have in common. Compare the failing or slow events with the rest: a route, a host, a customer, or a version that shows up much more often among them is a lead.
- Check deploy impact. Group the bad events by
resource.service.versionover a window that covers the deploy.