Archive

Everything published so far, newest first. Nothing is removed once it is up; corrections are appended rather than edited in.

Operations

The outage that was actually a clock

M. Rautenberg · 8 days ago

The first hour went into storage, because that is what the dashboards said. Write latency was up, the queue depth graph had the shape everyone recognises, and the storage team had a maintenance window open on an unrelated cluster. Three people spent an hour disproving a coincidence.

What actually happened was that two application hosts had drifted 400 milliseconds apart after an NTP source was quietly removed from the pool in a configuration change eleven days earlier. The lease logic treated a lease issued by the faster host as already expired, renewed it, and the slower host did the same in reverse. Neither host logged anything unusual, because from each one's point of view nothing unusual had happened.

The change we made afterwards was not better time synchronisation, although we did that too. It was to make the lease code log both clock readings whenever it rejects a lease. The next occurrence will be diagnosed in a minute rather than six hours, and that is worth more than the prevention we cannot guarantee.

Observability

Reading a flame graph you did not generate

S. Vogelsang · 31 days ago

Profiles arrive in tickets without context more often than anyone would like. The sampling interval is unstated, the workload is described as 'normal load', and the person who captured it has gone home. There is still usually something to be had.

Start at the bottom and read the width, not the height. Depth tells you about call structure, which is rarely the problem; width tells you where time went, which usually is. A frame occupying a third of the width is worth understanding even if you have no idea what it does, and the name is often enough to find out.

The common trap is treating a wide frame near the bottom as the culprit. It is an ancestor. Follow it upward until the width splits, and the split is where the interesting decision happens. If it never splits, you are looking at a single hot path and the answer is in that function.

Storage

A storage migration measured in deleted code

K. Pflueger · 44 days ago

The data moved in nine days with a script that fit on one screen. That was never going to be the hard part, and we knew it going in.

The hard part was eleven separate pieces of retry and workaround logic that had grown around the old store's behaviour over six years. Three of them compensated for a consistency model the old store had abandoned in 2021. Two retried an error code that no longer existed. One, memorably, slept for a fixed two seconds before every write, with a comment that said 'do not remove' and no further explanation.

We removed all eleven, which is the number we are actually proud of. The post-migration code is 1,400 lines shorter and the error rate is lower, because several of those workarounds had become the cause of the thing they were written to work around.

Build systems

Build caches lie by omission

A. Treichel · 58 days ago

Our cache key hashed every source file, the lockfile and the build configuration. It did not hash the compiler version, because the compiler version was pinned, and a pinned thing does not need hashing. That reasoning was correct until someone changed the pin.

What followed was a week of a binary that behaved differently on two machines whose caches had been populated on either side of the pin change. Both builds were reproducible. Both were cache hits. They produced different output, and every tool we had was telling us they could not possibly differ.

The rule we now apply is that the cache key covers everything that can change the output, including things that are pinned, because a pin is a value that can change rather than a constant. The key got longer and the hit ratio dropped by about four percent. That is the correct price.

Networking

Rate limits are a conversation, not a wall

N. Heidrich · 73 days ago

A rate limiter that returns 429 and nothing else is asking every client to guess. Most guess badly, and the popular guess is to retry immediately, which is how a rate limit becomes an outage.

Retry-After costs nothing to emit and changes client behaviour completely. Well-behaved clients wait exactly as long as you asked. Badly-behaved ones at least become easy to identify, because they are the ones ignoring a header you can see in your own logs.

The second thing worth emitting is the remaining budget on successful responses, not just on rejections. A client that can see itself approaching the limit can slow down before hitting it, which produces a much gentler load curve than the sawtooth you get from clients discovering the limit by being rejected.

Operations

The runbook nobody could follow

C. Obermeier · 91 days ago

The runbook was thorough, current and well organised. During a real incident, the engineer on call read four sentences of it and then started guessing, and when we asked afterwards they said it was faster to guess. They were right.

The documentation had been written to be complete, which meant every procedure opened with the conditions under which it applied. At three in the morning, that is three paragraphs between the reader and the first command. Completeness had been optimised at the expense of the only situation the document exists for.

The rewrite puts the commands first and the reasoning in a collapsed section underneath. Each procedure starts with what to type. It is a worse document to read end to end and a far better one to use, and the next incident ran twenty minutes shorter than the comparable one before it.