Every system we have built or inherited has failed in one of seven ways. We know because we keep a ledger of our own mistakes, and when we sorted it, the incidents collapsed into a small number of repeating shapes. This is that taxonomy, with the real incident behind each one.
The rule underneath all of it: assume it is wrong until it survives a challenge. Not because pessimism is a virtue, but because every one of these failures looked fine from the inside.
1. The green light that lied
A dashboard said healthy. The pipeline had been dead for weeks.
We discovered that a deployed build was an older snapshot that had silently reverted about thirty correctness fixes and dropped nine named data feeds. Zero runs. For weeks. The health badges stayed green the entire time, because the health check was checking that the service responded, not that the service had done anything.
The lesson: a status is not a fact until something outside the system confirms it. A health endpoint that returns 200 when the database is empty is not monitoring, it is decoration. Every health check we build now hits a data-backed endpoint and asserts on freshness, and the watchdog lives outside the thing it watches.
2. The number that was made up
This is the one that should frighten you most, because it is invisible.
We ran an adversarial pass over our own headline figures, with the explicit job of refuting them. Six of nine failed. Not "needed nuance." Failed. A count of fabricated records that existed nowhere in the repository. A pair of call-tracking numbers whose real data said the opposite. A dollar figure where two different metrics shared a name and nobody could say which one the number came from.
Every number that survived was traceable to a query, a commit, or a countable list. Every number that lived only in a summary document was wrong.
That is now the publication rule: if you cannot re-derive it from a query, a commit, or an enumerable list, it is not evidence. It is a rumour you wrote down.
The lesson: a confident wrong number destroys more trust than a missing one. Omit it, or say "varies."
3. The model that was cheating
We built three machine-learning models. They scored well. Then we attacked them.
The accuracy was an artifact. One had target leakage through mutable rows that carried the answer. Another had circular labels, where the thing being predicted was derived from the thing predicting it. Under adversarial review the performance collapsed toward noise.
We shipped none of them, flagged all three as untrusted, and kept them shadow-only.
The lesson: high accuracy is a symptom, not a result. A model that scores well on the first try deserves suspicion, not a launch date. Attack it before you believe it, because your client will not, and the model will be making decisions about their business either way.
4. The silent death
An enrichment job stopped producing new records. Nothing alerted. The job kept running, kept costing money, and kept writing nothing.
Separately, and worse: an approval console marked outage campaigns as sent without ever calling the mail vendor. Nothing went out. The audit trail said everything had.
The lesson: silent failure is the default state of an unattended pipeline. Not an edge case, the default. Anything unattended needs a yield check, not just an error check, because "no error" and "no output" look identical in a log. Our pipelines now monitor yield per batch and halt when it hits zero, and they page a human rather than writing a line nobody reads.
5. The overclaim, including by our own reviewers
We built an adversarial verification system to catch bad claims. Then the verification system produced a bad claim.
A verifier ran a search, the search was killed partway through, and it read the empty output as proof that the thing did not exist. It reported a finding as refuted when it had simply never finished looking.
The lesson, and it governs everything else here: reviewers overclaim too. Verify agent-claimed findings against the primary source, and never verify a number against the same snapshot that produced it. A truncated process that returns nothing is indistinguishable from a complete process that found nothing, so every negative result now has to be backed by a command that provably ran to completion, with a file count and an explicit end-of-scan marker.
There is a corollary we hold ourselves to. One question in that audit remains genuinely unresolved: a gutted repository where we could not confirm or refute whether code had been deleted. It is recorded as unknown, not as refuted. Unknown is an answer. Pretending otherwise is how item 2 happens.
6. The last mile that was never walked
Here is the number we least want to publish.
Across sixty-nine projects, fourteen were built and never launched. The code works. Nobody uses it.
They died on a single external dependency, every time. A DNS change the client never made. An IT team that never opened a port. A vendor API whose access was requested and never granted. In each case the engineering was finished and the thing sat there.
The lesson: a project is not done when the code works. It is done when a human uses it. We now identify the one external dependency that can strand an engagement, and we resolve it before writing the code. If the answer to "who has to do something outside this room for this to go live" is unclear, that question is the project until it is answered.
Twenty percent is a humbling rate. It is also, we suspect, an ordinary one that most firms simply never count.
7. The environment that ate the work
Less dramatic, more common than anyone admits. Work built in a syncing cloud folder that corrupted under concurrent access. Code deployed to production and never committed anywhere, which is the single worst state a system can be in: it is running, it is load-bearing, and it exists in exactly one place that nobody has a copy of.
The lesson: deployed-but-never-committed is worse than not shipped. If it runs and it is not in version control, it is not a system, it is a liability with uptime.
What the taxonomy is actually for
None of these are exotic. That is the point. They are the ordinary ways that a system which looks healthy is not, and every one of them is invisible from the inside, which is why they need an outside.
When we start an engagement, this list is the checklist. Is the health check checking data or just liveness. Can each headline number be re-derived from a query. Has anyone tried to break the model. Does the pipeline know the difference between no errors and no output. What single external dependency can strand this. Is anything running that is not in version control.
Six questions. They find things almost every time, and they cost a morning.
The uncomfortable part of publishing this is that most of these incidents are ours. We caught our own reverted deployment, our own fabricated statistics, our own models that were cheating, our own fourteen projects that never launched. We would rather write that down than repeat it, and a mistake that is fixed but not written down is a mistake you will make again.
PurviewX is embedded AI leadership for companies sitting on real operational data. We find out whether your AI actually works, including ours. Start a conversation.