Roughly two minutes after I killed a backfill pulling filings from an SEC data source, two Lambdas finished the same batch a couple milliseconds apart. This was unexpected considering the backfill was designed to run sequentially. I had each worker capped at eight requests per second against the ratelimit I was designing around.
It was a pretty standard backfill - the driver hands a batch to Lambda, waits for it to finish, then moves onto the next batch. With only one worker and the eight requests per second, everything should have stayed comfortably within the limit.
What could possibly go wrong?
The answer to that rests in the backfill I killed. You see, I didn’t actually kill the backfill - I killed the local driver that invoked the Lambda. Naturally, this didn’t kill the Lambda itself. As a result, after I restarted the driver there were two Lambdas running concurrently on the same batch.
The evidence was two milliseconds apart
1,000 SEC submission paths had been handed to both executions with the first run completing at
04:18:41.691
and the latter at
04:18:41.693.
Looking at the counters, the old worker reported 957 newly fetched documents with 43 already archived while the new showed 279 fetched and 721 already archived. They were watching each other work.
For a little under two minutes I had two independent Lambdas fetching from the SEC datasource capped at eight requests per second each. This meant that the ten requests per second hardcap I was designing around was now being exceeded by up to six excess requests per second.
Given that uniqueness was designed within the table ingesting the data itself and the SEC source didn’t throttle either of the workers, it didn’t actually break anything. Neither of the batches failed either meaning there was a bug that got silently swept under the rug.
As far as incidents go, it was a relatively boring one but one that served as a good example of how infra problems can sometimes slip by unnoticed.
The design had previously felt pretty safe
Duplicates were already factored in. The pipeline was built with retries and reprocessing accounted for. Fetching the same filing twice was designed to create only a single row. The archive path was deterministic and the loader had conflict handling.
As far as I was concerned, the pipeline was idempotent which was exactly what hid the bug. However, just because the result was safe to repeat didn’t mean that the work itself was.
When it comes to the database itself, it didn’t matter that the two Lambdas fetched the same document since only one row would actually be inserted.
However, considering it from the point of view of the SEC provider, I was running double the requests. The downstream invariant protected my storage but did nothing for the system on the other side of the network.
As obvious as it seems in retrospect, it was a lot less clear looking at the pipeline since duplicates were already accounted for.
There wasn't one thing called "the operation" that was either idempotent or not. There was a chain of side effects:
- start the worker
- make requests against an external API
- archive the responses
- load the records
- emit the new state
Making the last few safe didn't magically do the same for the first few.
Sequential code is not a concurrency control
The part I like least in retrospect is that the intended constraint was documented.
The backfill driver was sequential on purpose.
Parallel workers would multiply the per-process request rate, so the code ran one batch at a time. There was even a guard checking that the scheduled version of the same pipeline was disabled during the manual backfill.
That guard worked. But it was also irrelevant.
It watched the scheduled door. My duplicate worker came through a completely different one: a manually started process whose client disappeared while the remote execution continued.
Nothing violated the local control flow.
The first driver invoked one Lambda. The second driver invoked one Lambda. Each Lambda individually obeyed its rate limit.
Every component was doing what it had been designed to do and the system-level invariant was still false.
That's the more useful way I've started thinking about these bugs.
If something must never happen concurrently, "the caller is sequential" is not the same thing as "the operation cannot run concurrently."
The former is a property of one execution path. The latter needs an authority shared by every execution path. I had the first and had been mentally treating it like the second.
A timeout made the same mistake visible from the other direction
Later in the same backfill, the client hit a read timeout waiting for another batch.
From the driver's point of view, the result was straightforward. It hadn't received the artifacts, so it reported the batch as 1,000 short.
AWS told a different story though. All 1,000 documents had successfully loaded server-side. They finished loading roughly ten minutes before the client finally timed out.
Nothing was missing. The observer had stopped observing. That just made the underlying mistake harder to ignore:
a client side outcome is just a statement about what the client knows, not necessarily about what the system did.
- "Timed out" does not mean "failed."
- "Killed" does not mean "cancelled."
- "No response" does not mean "no side effect."
Once work crosses a process boundary, those words stop being synonyms.
So what actually fixes it?
The annoying answer is that there wasn't one obviously correct patch.
One option is to put a real lease around the work. Before a fetch worker proceeds, it acquires a shared lock in something like DynamoDB or the database. A second worker sees the lease and refuses to run.
That actually enforces the invariant. It also introduces lease duration, expiry, recovery and another dependency into every invocation.
Another option is an advisory preflight check. Before the local driver starts, look for evidence of a fetch that began without a terminal event. If one appears to still be alive, refuse to launch another.
That's much cheaper.
It also can't stop a worker that's already running. The race still exists but you're just making it harder to enter accidentally.
The cheapest route is barely a system at all. It's just documenting the rule.
If you kill the driver, wait one full Lambda timeout before restarting it.
That’s effectively what the system requires today.
It's also the kind of operational knowledge that works perfectly until the moment somebody needs it which tends to be the exact moment they’re debugging something else.
I haven't become allergic to those kinds of rules. Not every hazard deserves another distributed lock. But I do want to be precise about what each option buys.
- A runbook reduces the likelihood of a second worker.
- A preflight check detects evidence that one may exist.
- A lease makes concurrent execution something the system itself can refuse.
Those are three different guarantees. But calling all three "fixes" hides the interesting part.
Idempotent to who?
A lot of folks think about idempotency mostly from the perspective of the system receiving the result.
- Can I retry this insert?
- Can I replay this event?
- Can I rerun this batch without corrupting state?
Those are still the right questions. There's just another one hiding next to them:
What happens to every system I touch if there were suddenly two of me?
Sometimes the answer is boring.
- Two workers write the same deterministic object. Fine.
- Two loaders race for the same unique key. Fine.
- Two notification workers send the same email. Not fine.
- Two fetchers independently obey an eight-request-per-second limit against the same ten-request-per-second API. Also not fine.
Idempotency isn't a property you sprinkle over a workflow until it becomes safe to repeat. It belongs to individual effects, and a distributed operation can be repeatable on one side while doing something very different somewhere else.
In my case, the database was protected exactly the way I wanted but the upstream wasn't protected at all.
The near miss was useful because nothing failed loudly enough to hide that distinction.
The next time I say an operation is safe to retry, I want to be able to answer a more annoying question:
Which part?