Every site on this host shares one edge cache. The thing that keeps one customer's files from being served on another customer's address is a single string: the cache key. Ours looks like this:
https://cache.internal/{siteId}/{deployId}/{path}(A short suffix after the two ids can mark variants of the page's body — some site settings, or which kind of hostname asked; it is beside the point here.)
An earlier post about the two planes mentioned, in a paragraph, that deleting siteId from that key once left the whole test suite green. This is the long version: why it survived, whether siteId was even needed, the attack we measured when we went to find out — and a test that was supposed to stop that attack, which turned out to have been catching it only by accident until the week this was written.
Why the key is ours to get right
The Workers Cache API does not know which site a request is for. Its key is whatever Request or URL string the Worker passes in, and nothing else. The visitor's hostname is not part of it unless we put it there.
We deliberately do not. The bytes of one deploy are the same at every one of our own addresses it is served from, so those share one cache entry. (A customer's own domain gets a separately marked entry, because one widget is written into the page only on our hostnames — a difference in the body, not in who the site belongs to.) So the key names the content instead of the address: which site, which deploy, which file. A new deploy gets a new id, so the old entries simply stop being asked for. Publishing never needs a purge.
That makes the key the whole of tenant isolation at the cache layer. Leave the site out, and the only thing keeping two customers apart is whether their other identifiers happen to collide.
How the first mutant survived
Mutation testing is simple to describe: break the code on purpose, run the tests, and expect something to fail. If nothing does, the mutant survived, and the tests were not protecting what you thought.
The suite that guards this key starts two sites with the same path and different contents, requests both alternately, and asserts each gets its own bytes. When we removed siteId from the function that builds the key, every one of those assertions stayed green, for two reasons that compound:
- The expectations were built with the function under test. To check that an entry was stored, the tests called
cacheKeyUrl(siteId, deployId, path)— the same function the Worker uses. Break it, and the code and the expectation move together. - The fixtures used different deploy ids. Site A's deploy and site B's deploy were different strings, as they always are. With
siteIdgone, their keys were still different, so no collision ever happened, so no test ever saw one.
The fix in August was two assertions that spell the key out by hand, with a comment above them explaining that spelling the key out is the entire point. Those two tests check the shape of the key. They do not show what goes wrong without it.
Was the site id even needed?
This is the question worth taking seriously, because the honest first answer is "maybe not". Deploy ids are ULIDs. Two sites will never have the same one. If the deploy id in the key always came from our own records, a key without the site would never collide, and the mutant surviving would be a test quality problem with no consequence.
But the deploy id does not always come from our records. Every deploy has a preview address of the form {deployId}--{subdomain}, so an owner can look at a draft before it goes live. On that path the Worker looks up the subdomain — which fixes the site — and takes the deploy id from the hostname the visitor typed.
So we measured it. On a copy of the router outside the repository, with siteId removed from the key: publish site B, request one of its files on B's address so it is cached, then ask for the same file through site A's preview address, with B's deploy id in it.
https://beta1.dropto.live/read.js → 200 "file of B"
https://{B's deploy id}--alpha1.dropto.live/read.js → 200 "file of B"Site A's address served site B's file. Deploy ids are unique, but they are not secret — anyone who has been sent a preview link has one. With the real key the second request is a 404: the key carries A's site id, so the cache lookup misses, and the storage prefix — which also starts with A's site id — holds nothing for B's deploy.
Uniqueness was never the property that mattered. What mattered was who gets to choose the value. The site id comes from our lookup; the deploy id, on this path, comes from the request. A key with only the second in it is a key the visitor writes.
The test that named the attack
There was already a test for exactly this, added with the preview addresses in August: cannot reach site A's deploy through site B's preview hostname. With siteId removed, it failed. We would have counted the mutant as killed and moved on.
Then we ran it on its own, and it passed.
It turned out the test ran against an empty cache. A cold request falls through to storage, where the object prefix also starts with the site id from our lookup, finds nothing, and returns a 404. The cache key is never consulted. The test was a test of the storage prefix — a different rule, enforced in a different place.
In the full file, it failed only because of what the test before it left behind: that test opened site A's draft through A's own preview address, which left that draft in the cache. With the cache warm, the attack worked, and the assertion caught it. The mutant was being killed by test order.
The fix is the test it should have been: warm site A's draft first, then attack it from site B. It fails under the mutant whatever runs before it. We kept the old one, because the cold path is real and the prefix deserves its own test. Across the router's 930 tests, removing siteId from the key now fails 27, removing deployId fails 44, and removing both fails 64.
The guard that blocked the proof
One more thing made this harder than it should have been. This repository has a pre-edit hook that refuses an edit to the code building the cache key when the new text leaves out siteId or deployId. That is the point of it.
It also refuses the edit a mutation test needs to make. Twice, running the mutation procedure meant a person switching the guard off by hand. The answer we settled on is a declared exception — a short note saying what is being broken and why, which opens the cache and header rules for thirty minutes and is logged — or doing the mutation on a copy outside the repository, as this post's measurements were. A guard and the procedure that proves its test still works should not be pointed at each other, but by default they are.
What generalises
None of this is specific to caches:
- Spell out at least one expected value by hand. The earlier post makes the general point; the practical form is one literal string per invariant that nobody is allowed to refactor onto a helper.
- Random, distinct fixtures hide collisions. If the bug is "two things share a key", a fixture where every id is different will never show it.
- Run a killing test on its own. A mutant that only dies in the full suite may be dying of another test's leftovers.
- Test each layer that can refuse. A cold cache and a warm one take different paths, and each path has its own reason to say no.
- Unique is not the same as authoritative. An identifier that a request can supply is an input, not a guarantee.