Telemetry Machine blog

Thanks for Your Attention

Jack Popov — Mon, 02 Nov 2026 00:00:00 GMT

Hello! Thanks for giving me some of your attention. This post is a little meta, because it's all about how you get someone's attention.

Why do I want your attention? Because I just launched my observability startup, Telemetry Machine, and now I'm trying to do the seemingly impossible: getting noticed in the age of AI slop.

I want to talk about another place where you have to capture and hold attention despite tremendous competition: street performance (aka busking).

Recently, I found myself captivated by a busker. What captivated me wasn't his tricks. It was how he got people to notice and stay.

My kid and I were on our way to the Transport Museum in Covent Garden, which is famous for its street performances. I had never stopped for one, because I think of them as extremely touristy. We had already walked past a magician.

But then we heard yelling. I almost thought something violent was happening. My kid seemed curious, so I asked if he wanted to check it out. We walked inside to a busker's pitch, a performing spot with three benches that has apparently been there for decades. A small man in a funny hat, dressed almost like a leprechaun, was walking around and screaming at people. I didn't realize it yet, but that screaming was very purposeful: it had just brought us over.

We started to walk past, noncommittal. But then he looked me straight in the face and screamed, "STOP!" Of course I stopped on a dime. What was I going to do? Ignore him? (I would be a great victim for hypnosis.)

He was just getting started, with almost no one around him. This is the hardest part of the show: working up the initial audience.

About five minutes in, he had maybe ten people watching. He said something like: "Okay, I'm going to do a trick for you guys, but I want you to make lots of noise and clap, because people will hear you, think I'm really good and walk over. And if we get a big audience, I'll feed off your energy and give you a really amazing show." He walked us through how to clap and cheer, which itself made a huge amount of noise.

Then he did the trick. It wasn't anything major, but it was obviously not something we could have done. He did a handstand on a bench and rolled forward into a hat. But he did this with lots of showmanship and noise. We clapped and cheered.

He had basically enrolled us in building his audience. He could have simply said, "Hey, I'm about to do a trick, please clap." What he said instead was much more convincing. Why should we clap? Straight up, because if we clap, it will be loud and people will come over. And we did want the audience to be big, because if we're sitting there and nobody else is around, it's kind of boring.

Getting the first few people to make noise, I found out later, is a well-known tool of buskers.

Then he built up his "finale". I say "finale" in quotes because, really, it was his only significant trick. He talked about how dangerous it was and what would happen if he fell.

Before he did it, he gave us what I later learned is called the hat line, where he said this is my living, this is how I feed my family, please pay for it. He spoke about that for maybe five minutes. I wonder if there's psychology to this. Five minutes in, we hadn't left. So we were showing ourselves, and each other, that we'd pay.

Then came the finale: a handstand, really high up and precarious. Everybody clapped. We were impressed, I think partly because we had spent 30 minutes investing ourselves in it.

As soon as it was over, a couple of people walked away, but it felt like almost everyone, 30-40 people by now, got up to pay. The next performer was already waiting, geared up and miked.

Watching a master at their craft is particularly enjoyable. In this case, the craft wasn't acrobatics. It was crowd work: understanding how to capture and manage our attention.

It also humbled me in my own quest to find my first customers. Buskers start from nothing every single performance.

Now, I'm not going to talk about what I can learn about sales from this, because that would go way over my threshold for LinkedIn cringe. But what we're doing is not so different from busking. We're giving people something we think they want or need, and asking for money in return. The hard part is getting attention in the first place.

Right now, Telemetry Machine is at the almost-no-audience stage. We're working with our first few potential customers. (Early customers, I've learned, are absolute angels.)

Will we ever become masters at this, like that busker? Well hey, why don't you sign up to our blog and find out?

The New State of the Art in Observability

Jack Popov — Fri, 28 Aug 2026 00:00:00 GMT

Observability databases have undergone a revolution in the past five years. I don't think most people have noticed this. I noticed, of course, because I've been working in observability for years and saw these lessons slowly creep in. Now I see it everywhere.

In short, a new state of the art has formed in observability.

In this post, I'll take you from a traditional observability architecture – Prometheus – and show you how a modern observability system is profoundly different.

This architecture is now widely used. It's used by ClickHouse (SharedMergeTree), Honeycomb, Datadog (Husky), Axiom (EventDB), InfluxDB (IOx), Loki, Elastic Serverless, and of course here at Telemetry Machine.

I'll spoil it all now: the big secrets are object storage (like S3) and separation of storage and compute (unlinking your database processes from their disks). Everything else is details. And the details are very juicy.

We applied all these lessons when making Telemetry Machine. For us, the goal was to make it as cheap as possible. If you're curious why we believe that observability wants to be cheap, read our big manifesto.

So, what changed? Let's start with the grand old lady of observability, Prometheus.

Let's start with a disk

The simplest way I know to picture a traditional observability database is a standalone Prometheus server.

Data comes in. Prometheus writes it to a disk attached to the server. When you make a query, the same server reads that disk and answers you.

The stateful storage processes are tied to particular disks. Once a machine is no longer enough, you must expand or repartition the data.

Now, imagine if you suddenly get a giant spike in traffic. This is commonplace in observability. For example, if a large consumer app releases a bug, the amount you are ingesting could suddenly multiply dramatically.

If you're lucky, you may be able to vertically scale the VM or pod. But horizontal scaling is difficult, because the processes are tied to the disks, and you will likely need to physically move data (e.g. if adding a new shard).

And, of course, when the spike dies down, you have to undo that work.

What can we do?

The answer is separation of storage and compute

What if the data did not belong to any server or process?

That is, your processes that write and read the data have no actual ownership to the data.

The common way to accomplish this is via object storage like S3. The place your observability data permanently lives is no longer a disk tied to your database processes.

What does that buy us?

First, storage capacity is no longer tied to your servers. If you want more storage, you do not need to change disks or migrate anything.

Second, you get horizontal scaling. If you need to suddenly scale up your ingestion a lot, you can do so easily, because they are now stateless.

Third, you get cost savings. Traditionally, observability databases use SSDs so they can be fast. That's very expensive. Object storage is dramatically cheaper.

But, so far we have merely said "separate storage and compute". What does it actually concretely look like?

Let's make the architecture concrete

Fundamentally, an observability database has three jobs.

It takes in data (ingestion). It answers questions about that data (querying). It maintains the data.

The modern architecture gives those jobs to different pools of processes:

Writers read from a durable write-ahead log (at Telemetry Machine, we just use Kafka. We throw data into Kafka as early as possible). They collect records in batches, turn the batches into queryable files, and upload them to object storage.
Query workers find the objects relevant to a query, read the necessary objects in parallel, and combine the results. We use DataFusion.
Maintainers tidy up the data: they merge small files (more on that later), delete old data, prepare indexes.

This is what it looks like at Telemetry Machine:

OTel exporters
      │
      ▼
API processes ──► Kafka ──► writers ── upload Parquet ──► object storage (R2)
                              │                               │          ▲
               register files │                    read files │          │ merge files
                              ▼                               ▼          │
                           catalog ── visible file set ──► query workers │
                              ▲                          (DataFusion)    │
                              │                                          │
                              └──────── update catalog ──── maintainers ─┘

We call it EventLake. Lake is the important word. Under the hood, it's all a data lake, but tuned for the constraints and non-constraints of observability data.

What do we put in object storage?

Let's focus on the data itself. For observability (and all analytical data, really), columnar storage is the ace up our sleeve. Because observability data is extremely repetitive (e.g. repeating the same log statement over and over), it compresses extremely well.

Columnar storage can be used for both old and new-style observability architecture, so we don't go deep on it here. For the best explanation of columnar storage, see Martin Kleppmann's Designing Data-Intensive Applications. For a public blog post, see this explanation from MotherDuck.

What is important in our context is how we treat this columnar data: when a writer process writes a batch, it creates one file¹ and that file is immutable. This immutability is what enables concurrent writes and reads by stateless processes, because they never need to coordinate an in-place update to a file.

A common format for such files is Parquet (used by us and InfluxDB, for example), but databases often write their own formats. Parquet is a great fit for object storage, since each file is internally self-describing and queryable by itself. For example, a Parquet file already has the metadata necessary for predicate and projection pushdown (i.e. queries only need to read the actual bytes relevant to the query, meaning we don't need to read the entire file). This is particularly useful when these files are stored in object storage, as object storage calls are slow.

This is where the "non-constraints" of observability come in handy. Log files, traces, metrics, etc very rarely need to be mutated (only if you, e.g. want to rename an OTel attribute or if a type changes). The trade-off of immutable files is that if you do need to mutate them, you need a lot of extra machinery, but this machinery is only rarely invoked for observability data.

How does the database know which files exist?

We now have a pile of immutable objects in S3. How does a query know which ones belong to the database?

A good analogy is a filesystem like ext4. A filesystem is not merely the contents of your files. Every filesystem also has a little embedded database for tracking said files.

Similarly, for observability data, we use a catalog.

For example, here's some of what we store in the catalog for each Parquet file.

Metadata	What it lets us do
R2 URI	Find the file in object storage.
Min/max timestamp	Skip files outside a query's time range.
Column min/max	Skip row groups that cannot match a query.
Partition	Narrow the query to the relevant part of the dataset.

There is one aspect of the catalog which is crucially important, and core to what makes this architecture work: consistency is isolated to metadata. The consistency of the Parquet object is not important. The catalog decides which of those objects is actually visible to queries. For example, if a write must be retried, and an object is uploaded twice, there is no double-counting in a query as the catalog records the file only once.

In our case, the catalog is simply Postgres.

The small files problem

Of course, it's not all fun and games. One of the major bugbears that must be dealt with in this architecture is the small file problem.

Remember, every time a writer reads a batch from the WAL, it uploads a file to object storage. Reasonably, this happens every few minutes. Given that you will run many writers (e.g. sharded across customer datasets), this means thousands of files per minute are created.

You can't let these files build up. For example, each file is a row in your strongly consistent database, which is expensive. It also means queries are slower.

The maintenance job that fixes this is compaction. Essentially, a maintainer reads in multiple files, merges the data, and writes out one big file. This is where the consistency of the catalog is crucial: to avoid double-counting data in queries, the old files must be retired and the new file activated atomically.

Naturally, each maintainer is stateless, letting it be scaled independently of ingestion and query processes.

Object storage is slow

If you have used Thanos, you would have noticed that queries against Thanos (which go out to object storage) are much slower than regular Prometheus queries. The fact is, object storage means data is served over the network – even over the internet – and is much slower than local SSDs. This is one of the major trade-offs of the new architecture.

Thankfully, there are ways to alleviate the slowness.

First of all, both R2 and S3 support ranged reads. This means you can download only the parts of a file that actually serve the query.

Second, requests to an object store can be parallelized. Parquet stores row groups and column chunks at known byte offsets, so workers can fetch the required ranges concurrently. If you think about it, whereas a traditional database might be limited by its local disk, object storage makes the network your main limit. This means that for a very large query, if your network interface is not saturated, you may be leaving performance on the floor.

Third, you can make use of two non-constraints: one, data that was recently accessed is more likely to be accessed again. This means that once you download data from the object store, you can cache it on a local disk. This is a trade-off you can make of performance vs cost. (You may think of it a little bit like the Linux page cache, but instead of caching disk data in memory, you are caching network data on disk. In fact, internally we use something like the Linux LRU algorithm.) Second, in observability recent data is much more valuable than older data. This means that recent data can be prefetched. Again, this lets us trade off cost versus performance.

Object storage is not magic; it has trade-offs. That said, the dramatic cost difference between it and the disks usually recommended for observability² makes dealing with the trade-offs worth it.

Putting it together

Let's recap the old versus the new architecture.

In the old world, the ingestion and query processes of a database ran on the same server, tied inextricably to the same disk. They were difficult to scale, it was difficult to deal with low disk space, and the architecture required expensive disks.

In the state of the art, ingestion and query processes are stateless and independent from each other. They are similarly independent from the data, which is stored on remote disks, usually in object storage.

There are trade-offs, mostly related to performance, but these can be largely alleviated. The cost gains are significant, and worth it.

P.S. Use Telemetry Machine

Observability is expensive, and the cost to run the infrastructure is only one piece of it. This architecture helps dramatically, but it's not enough. We believe that observability is crucially important, and for that it must be widely adopted. One of the major blocks to wider use of observability is that it's too damn expensive. So, we made it cheap. Feel free to check us out at telemetrymachine.com or read why we believe observability doesn't have to be expensive.

A writer may produce several files from one batch. This does not change the architecture: every output file is immutable. ↩
ClickHouse's self-hosted hardware guide says that for maximum performance it recommends “directly attaching provisioned IOPS SSD volumes from AWS.” See Sizing and hardware recommendations. ↩

Observability Doesn't Have to Be Expensive

Jack Popov — Fri, 12 Jun 2026 00:00:00 GMT

As software engineers, we learn many best practices. A few of these are non-negotiable: software that follows these practices is strictly better than software that doesn't. For example, version control, unit testing, and observability.

Without version control, collaboration is impossible; without unit testing, software is buggy. Without observability, we can't tell if our users are hitting errors or slowness, or whether our service is completely down.

Unfortunately, there's one big difference between observability and the other best practices: it's orders of magnitude more expensive. Thus, it's not nearly as widely adopted. Many projects avoid observability entirely. Even if a team does use it, they often operate in a failure mode: aggressive sampling, short retention, or turning off features.

Why is it so expensive? Some vendors claim that it's because modern software is too complex, while others say it's because observability itself is expensive to operate. This paints an extremely incomplete picture.

The truth is, almost every observability company is tuned to serve the biggest companies. This is only rational: it is widely assumed that big customers are the best customers. But serving big customers changes you. You hire salespeople, account managers, and many (many) engineers to win big accounts and squeeze as much revenue from them as possible. Inevitably, high prices follow.

It doesn't have to be that way. There are thousands of serious engineering teams that need observability but not the big company sales dance. A business built to serve them is fundamentally different from existing observability companies: lean, efficient, and far less expensive to operate. There is no need to charge extraordinary prices to build a sustainable observability company.

Every service, every Kubernetes cluster, every server should be observable. We've been told that's too expensive. It doesn't have to be.

Why observability loves big customers

Observability companies skew towards big customers for the same reason all B2B businesses do.

It is no secret that most software companies optimize for revenue growth. This is especially true if they take investment.

When you optimize for revenue growth, it's frequently advised to seek big customers. To name just a few reasons:

A big customer can bring as much revenue as 100 smaller ones, but is not 100x more difficult to win.
They have lower churn, e.g., because they are less likely to fail.
They have more opportunity for account expansion.

Consider Datadog. When they filed their S-1 to go public, customers spending more than $100k a year already represented 48% of ARR. By 2025, that had grown to 90%.¹ Clearly, the money is in big customers.

Serving big companies changes you

It's not enough to simply want big companies. You have to shape your company around them.

To win big customers, you need salespeople. If someone is going to commit $100k a year, they want to talk to you. They want a contract. They want to know you can handle security review, compliance, support. They want to know you will cater to their needs.

Since each large customer represents a meaningful chunk of revenue, losing one hurts. You need account managers and customer success staff, too.

This means building out a large sales and marketing organization. In 2019, 39% of Datadog's employees were in sales and marketing, slightly more than in R&D.²

That isn't all. Once a company is oriented around large customers, it is not enough to win those customers. Each account must be expanded. This is especially important if you need to grow fast and/or have investors.

A great way to grow a customer is to build more features and charge for them. Because the customer is big, they likely have a need for those features, and they likely have the money to spend.

This again takes a lot of people. In 2019, Datadog had 462 R&D employees.²

Of course, those engineers do a lot more than build enterprise features. But once a company is built around large accounts, the roadmap bends towards serving them. The core product inevitably suffers.

This is the standard B2B SaaS playbook. And it worked: Datadog's market cap of tens of billions of dollars is eye-watering. The observability market supports many other successful companies, seemingly all of them targeting the biggest customers.

Unfortunately, that leaves most of the market rationing telemetry, under-instrumenting, or priced out entirely.

An observability company for the rest of us

What does a company built to serve the rest of us look like?

First of all, it is lean. As we have seen, large teams lead to large costs.

Second, it is efficient. Find ways to get infrastructure costs down.

Third, it is focused. Nail the core observability features we all need.

Finally, shape it around self-serve customers. Make pricing transparent. Write great documentation. Let people buy without excruciating sales calls.

Such a company can solve the core promise of observability without being sucked down the enterprise pricing gravity well.

Observability should be a no-brainer

With all the discussion of observability's high price, something gets forgotten: observability is great.

Who doesn't want to sleep well at night knowing their service is healthy?

All software benefits from observability, and we all benefit when software runs well. We should want observability to be widely adopted.

We've been told that's too expensive.

It doesn't have to be.

Datadog, Inc., 2025 Form 10-K. Used for the 2025 customer count and ARR concentration. ↩
Datadog, Inc., Form S-1/A filed September 9, 2019. Used for the 2019 customer count, ARR concentration, S&M/R&D headcount, and quoted expansion strategy. ↩ ↩²