The New State of the Art in Observability

Observability databases have undergone a revolution in the past five years. I don't think most people have noticed this. I noticed, of course, because I've been working in observability for years and saw these lessons slowly creep in. Now I see it everywhere.

In short, a new state of the art has formed in observability.

In this post, I'll take you from a traditional observability architecture – Prometheus – and show you how a modern observability system is profoundly different.

This architecture is now widely used. It's used by ClickHouse (SharedMergeTree), Honeycomb, Datadog (Husky), Axiom (EventDB), InfluxDB (IOx), Loki, Elastic Serverless, and of course here at Telemetry Machine.

I'll spoil it all now: the big secrets are object storage (like S3) and separation of storage and compute (unlinking your database processes from their disks). Everything else is details. And the details are very juicy.

We applied all these lessons when making Telemetry Machine. For us, the goal was to make it as cheap as possible. If you're curious why we believe that observability wants to be cheap, read our big manifesto.

So, what changed? Let's start with the grand old lady of observability, Prometheus.

Let's start with a disk

The simplest way I know to picture a traditional observability database is a standalone Prometheus server.

Data comes in. Prometheus writes it to a disk attached to the server. When you make a query, the same server reads that disk and answers you.

The stateful storage processes are tied to particular disks. Once a machine is no longer enough, you must expand or repartition the data.

Now, imagine if you suddenly get a giant spike in traffic. This is commonplace in observability. For example, if a large consumer app releases a bug, the amount you are ingesting could suddenly multiply dramatically.

If you're lucky, you may be able to vertically scale the VM or pod. But horizontal scaling is difficult, because the processes are tied to the disks, and you will likely need to physically move data (e.g. if adding a new shard).

And, of course, when the spike dies down, you have to undo that work.

What can we do?

The answer is separation of storage and compute

What if the data did not belong to any server or process?

That is, your processes that write and read the data have no actual ownership to the data.

The common way to accomplish this is via object storage like S3. The place your observability data permanently lives is no longer a disk tied to your database processes.

What does that buy us?

First, storage capacity is no longer tied to your servers. If you want more storage, you do not need to change disks or migrate anything.

Second, you get horizontal scaling. If you need to suddenly scale up your ingestion a lot, you can do so easily, because they are now stateless.

Third, you get cost savings. Traditionally, observability databases use SSDs so they can be fast. That's very expensive. Object storage is dramatically cheaper.

But, so far we have merely said "separate storage and compute". What does it actually concretely look like?

Let's make the architecture concrete

Fundamentally, an observability database has three jobs.

It takes in data (ingestion). It answers questions about that data (querying). It maintains the data.

The modern architecture gives those jobs to different pools of processes:

1. Writers read from a durable write-ahead log (at Telemetry Machine, we just use Kafka. We throw data into Kafka as early as possible). They collect records in batches, turn the batches into queryable files, and upload them to object storage.
2. Query workers find the objects relevant to a query, read the necessary objects in parallel, and combine the results. We use DataFusion.
3. Maintainers tidy up the data: they merge small files (more on that later), delete old data, prepare indexes.

This is what it looks like at Telemetry Machine:

OTel exporters
      │
      ▼
API processes ──► Kafka ──► writers ── upload Parquet ──► object storage (R2)
                              │                               │          ▲
               register files │                    read files │          │ merge files
                              ▼                               ▼          │
                           catalog ── visible file set ──► query workers │
                              ▲                          (DataFusion)    │
                              │                                          │
                              └──────── update catalog ──── maintainers ─┘

We call it EventLake. Lake is the important word. Under the hood, it's all a data lake, but tuned for the constraints and non-constraints of observability data.

What do we put in object storage?

Let's focus on the data itself. For observability (and all analytical data, really), columnar storage is the ace up our sleeve. Because observability data is extremely repetitive (e.g. repeating the same log statement over and over), it compresses extremely well.

Columnar storage can be used for both old and new-style observability architecture, so we don't go deep on it here. For the best explanation of columnar storage, see Martin Kleppmann's Designing Data-Intensive Applications. For a public blog post, see this explanation from MotherDuck.

What is important in our context is how we treat this columnar data: when a writer process writes a batch, it creates one file¹ and that file is immutable. This immutability is what enables concurrent writes and reads by stateless processes, because they never need to coordinate an in-place update to a file.

A common format for such files is Parquet (used by us and InfluxDB, for example), but databases often write their own formats. Parquet is a great fit for object storage, since each file is internally self-describing and queryable by itself. For example, a Parquet file already has the metadata necessary for predicate and projection pushdown (i.e. queries only need to read the actual bytes relevant to the query, meaning we don't need to read the entire file). This is particularly useful when these files are stored in object storage, as object storage calls are slow.

This is where the "non-constraints" of observability come in handy. Log files, traces, metrics, etc very rarely need to be mutated (only if you, e.g. want to rename an OTel attribute or if a type changes). The trade-off of immutable files is that if you do need to mutate them, you need a lot of extra machinery, but this machinery is only rarely invoked for observability data.

How does the database know which files exist?

We now have a pile of immutable objects in S3. How does a query know which ones belong to the database?

A good analogy is a filesystem like ext4. A filesystem is not merely the contents of your files. Every filesystem also has a little embedded database for tracking said files.

Similarly, for observability data, we use a catalog.

For example, here's some of what we store in the catalog for each Parquet file.

Metadata	What it lets us do
R2 URI
Min/max timestamp	Skip files outside a query's time range.
Column min/max	Skip row groups that cannot match a query.
Partition	Narrow the query to the relevant part of the dataset.

There is one aspect of the catalog which is crucially important, and core to what makes this architecture work: consistency is isolated to metadata. The consistency of the Parquet object is not important. The catalog decides which of those objects is actually visible to queries. For example, if a write must be retried, and an object is uploaded twice, there is no double-counting in a query as the catalog records the file only once.

In our case, the catalog is simply Postgres.

The small files problem

Of course, it's not all fun and games. One of the major bugbears that must be dealt with in this architecture is the small file problem.

Remember, every time a writer reads a batch from the WAL, it uploads a file to object storage. Reasonably, this happens every few minutes. Given that you will run many writers (e.g. sharded across customer datasets), this means thousands of files per minute are created.

You can't let these files build up. For example, each file is a row in your strongly consistent database, which is expensive. It also means queries are slower.

The maintenance job that fixes this is compaction. Essentially, a maintainer reads in multiple files, merges the data, and writes out one big file. This is where the consistency of the catalog is crucial: to avoid double-counting data in queries, the old files must be retired and the new file activated atomically.

Naturally, each maintainer is stateless, letting it be scaled independently of ingestion and query processes.

Object storage is slow

If you have used Thanos, you would have noticed that queries against Thanos (which go out to object storage) are much slower than regular Prometheus queries. The fact is, object storage means data is served over the network – even over the internet – and is much slower than local SSDs. This is one of the major trade-offs of the new architecture.

Thankfully, there are ways to alleviate the slowness.

First of all, both R2 and S3 support ranged reads. This means you can download only the parts of a file that actually serve the query.

Second, requests to an object store can be parallelized. Parquet stores row groups and column chunks at known byte offsets, so workers can fetch the required ranges concurrently. If you think about it, whereas a traditional database might be limited by its local disk, object storage makes the network your main limit. This means that for a very large query, if your network interface is not saturated, you may be leaving performance on the floor.

Third, you can make use of two non-constraints: one, data that was recently accessed is more likely to be accessed again. This means that once you download data from the object store, you can cache it on a local disk. This is a trade-off you can make of performance vs cost. (You may think of it a little bit like the Linux page cache, but instead of caching disk data in memory, you are caching network data on disk. In fact, internally we use something like the Linux LRU algorithm.) Second, in observability recent data is much more valuable than older data. This means that recent data can be prefetched. Again, this lets us trade off cost versus performance.

Object storage is not magic; it has trade-offs. That said, the dramatic cost difference between it and the disks usually recommended for observability² makes dealing with the trade-offs worth it.

Putting it together

Let's recap the old versus the new architecture.

In the old world, the ingestion and query processes of a database ran on the same server, tied inextricably to the same disk. They were difficult to scale, it was difficult to deal with low disk space, and the architecture required expensive disks.

In the state of the art, ingestion and query processes are stateless and independent from each other. They are similarly independent from the data, which is stored on remote disks, usually in object storage.

There are trade-offs, mostly related to performance, but these can be largely alleviated. The cost gains are significant, and worth it.

P.S. Use Telemetry Machine

Observability is expensive, and the cost to run the infrastructure is only one piece of it. This architecture helps dramatically, but it's not enough. We believe that observability is crucially important, and for that it must be widely adopted. One of the major blocks to wider use of observability is that it's too damn expensive. So, we made it cheap. Feel free to check us out at telemetrymachine.com or read why we believe observability doesn't have to be expensive.

Footnotes

1. A writer may produce several files from one batch. This does not change the architecture: every output file is immutable. ↩

2. ClickHouse's self-hosted hardware guide says that for maximum performance it recommends “directly attaching provisioned IOPS SSD volumes from AWS.” See Sizing and hardware recommendations. ↩