AI Strategy & Roadmapping · Cornerstone
The pattern is consistent enough to be useful. What separates AI pilots that ship into production from the ones that quietly die six months in.
Quantum Associates — Quantum Associates
· 14 min read
Run an autopsy on a stalled enterprise AI pilot and the same handful of failure modes show up, almost regardless of industry or organisation size. This is not because AI is hard (although it is). It is because AI exposes pre-existing organisational dysfunctions that traditional software projects can paper over. The pilot doesn’t fail because the model wasn’t good enough; it fails because the engagement was set up in a way that made shipping into production almost impossible from week one.
This piece names the patterns. Each one is recoverable from once you see it, but the recovery is much cheaper if the pattern is spotted in the first two weeks rather than at month six.
This is the most common failure pattern and the most diagnostic of the rest. A data scientist or external consultant builds an impressive proof-of-concept on a sample dataset. It demos well. The executive sponsor is enthused. The roadmap to production is approved.
What was missing from the workshop: the person whose pager goes off when the system breaks at 2am on Saturday. The customer-service team member who would actually use the recommendations. The accountant who would sign off on the AI-generated invoice automation. The compliance officer who would have to defend the system in an audit.
The pilot proceeds without their input. Six months in, deployment stalls — sometimes at the security review (the operator’s CISO never saw the design), sometimes at change management (the operator’s team rejects the rollout), sometimes simply because nobody on the operations side has the bandwidth to integrate the new tool into existing workflows.
The fix is mechanical and cheap if applied early. The operator who would run the system sits in the design workshop in week one. Their input shapes the architecture, the success criteria, and the integration plan. The pilot scope reflects what they can actually adopt within their team’s capacity. We invariably surface several uncomfortable truths — adoption blockers, integration gaps, unmet expectations — during these workshops; surfacing them in week one is what saves the engagement at month six.
Watch the language closely on a kickoff call. If the engagement is described as “build an AI-powered triage system” or “implement an LLM-driven contract review tool,” the pilot is already at risk. If it is described as “reduce average claim-handling time from X to Y over Z weeks” or “cut the legal team’s contract-review backlog by 40%,” the pilot has a fighting chance.
The distinction matters because the build-focused framing routinely produces systems that work but that nobody measures. The measurable-outcome framing forces the conversation about success criteria before any code is written. It also forces the harder conversation about whether the proposed system can plausibly move the number — sometimes the answer is no, and the engagement scope changes before any money is spent.
The fix is to refuse to start construction without an agreed measurement plan. The success metric, the baseline, the target, the measurement methodology, the dashboard the operator will watch — all agreed before week one. In our methodology this is part of the Design phase output; engagements that skip it become engagements that ship something and then ask “did it work?” months later, often without a clear answer.
A surprising number of AI pilots stall at week three when the engineering team realises the data the pilot needs is in a system nobody on the build team has read access to. Or that getting read access requires a six-week security review process. Or that the data exists but is so dirty that it would take a month of cleaning before any model could be trained or any RAG corpus indexed.
The pattern almost always traces back to an executive who confidently said “we have the data” in the kickoff workshop without consulting the data team. The data team would have said “we have the data, but accessing it through normal channels takes 6+ weeks, and the cleaning effort is substantial.” The pilot scope and timeline would have been very different.
The fix is to test data access in week one, not week three. Before any AI architecture is designed, the build team needs to confirm: which system holds the data, who owns it, how clean is it, how long does access take. If access is going to take six weeks, that gets factored into the pilot scope (sandbox export with a smaller subset, accept the delay, or pick a different use case). Discovering the access reality at week three turns a 12-week pilot into a 20-week pilot — and 20-week pilots are the ones that quietly die.
Look at the engagement letter. If somewhere in the deliverables it says “and operationalise the solution” or “with handover to internal operations,” the pilot is at risk. “Operationalisation” is doing an enormous amount of work in those phrases — it’s implicitly covering security review, change management, training, runbooks, integration with monitoring, on-call rotation, ongoing model evaluation, drift detection, and the dozen other practical realities of running an AI system in production.
The pilot team is rarely scoped or paid to do any of that work. The default-collapsed pattern: the pilot ships the technical system, declares operationalisation handover-ready, and the internal team is left holding a system they don’t know how to run.
The fix is to scope operationalisation as its own engagement phase with its own deliverables. Runbooks, training materials, observability handover, on-call documentation. In our Outcomes Method this is the Operate phase, and it has explicit pass criteria — the operator can complete the runbook unaided, the system has been observed running for a defined period without our intervention, the drift-detection alerts route to the right humans. Without those pass criteria, “operationalisation” becomes a euphemism for “throwing the system over the wall.”
A specific subset of Pattern 4. The pilot ships a system that works on the demo cases the team manually tested. There’s no automated evaluation harness, no labelled dataset to test future model changes against, no continuous quality monitoring in production. Three months later, the foundation model provider releases a new version, the model behaviour subtly shifts, and nobody notices for weeks.
This pattern is particularly costly because the symptoms are silent. The system keeps responding; the quality of the responses drifts. Customers complain in scattered, hard-to-aggregate ways. The team eventually notices but can’t quickly diagnose what changed because the evaluation infrastructure that would let them compare new behaviour to old never existed.
The fix is to build the evaluation harness alongside the system, not after. A labelled dataset of representative inputs with expected outputs. Automated evaluation that runs on every change. Production observability that traces every model call. Drift detection that fires alerts when behaviour moves beyond a defined tolerance. We treat these as Day-One requirements for any production AI system — not Phase 2 polish items.
This one is uncomfortable to talk about because it implies fragility, but the pattern is real. A passionate AI champion inside the organisation gets the pilot funded, picks the team, manages the vendor relationship, and is the system’s internal advocate. Six months in, they take a new role. Their successor doesn’t have the same investment in the project. Funding for ongoing maintenance gets cut. The system goes into slow decay.
This isn’t avoidable by selecting “more committed” champions; people leave roles regardless of commitment. It is avoidable by ensuring the AI system has institutional ownership independent of the champion. Documentation that another senior person can read and understand. Reporting that goes up the line to multiple stakeholders. Outcomes that show up on dashboards executives watch routinely. A budget line owned by a function rather than an individual.
We design for this from the engagement’s kickoff. The output of every engagement includes governance artefacts that survive personnel change. The handover process includes a deliberate “second pair of eyes” briefing for an executive who isn’t the original champion.
The opposite of the data-access surprise. Things go well in week two. The executive sponsor asks “while you’re here, can you also…” The team agrees. Scope expands. The original deliverables slip. Six weeks later, the original deliverables haven’t shipped, the expanded scope hasn’t shipped either, and the pilot is in trouble for both reasons.
The fix is to be ruthless about scope discipline. In-flight changes get evaluated against the agreed success criteria; if they don’t move the agreed metric, they don’t make the build. New scope ideas become candidate Phase 2 work — captured in a backlog, evaluated when the current phase ships, not bolted onto a running pilot.
In our Outcomes Method the decision gates between phases are the mechanism — scope conversations happen at gates, not in mid-phase. The discipline is uncomfortable in the moment and pays back enormously over the engagement.
The model works, ships into production, gets used. The first invoice from the foundation model provider arrives. It’s 3x what was forecast in the business case. CFO scrutiny intensifies. Within a few months, the project is reframed as “too expensive to scale” and quietly throttled or shut down.
The pattern stems from a single oversight at the Design phase: the business case included the build cost but didn’t adequately model the twelve-month run cost. AI inference costs scale with usage in non-linear ways — prompt length, output length, retrieval volume, context caching savings (or lack thereof), tier-based pricing changes, all interact. The math isn’t hard but it isn’t intuitive, and most pilot budgets don’t include it.
The fix is to require two cost models at the Design gate: the build cost (what we’ll bill you for the engagement) and the twelve-month run cost (a usage-based forecast with a confidence band). Both signed off before any build work starts. We refuse to advance past the Design phase without both — it’s a quiet protection against the most common reason successful pilots get killed at month nine.
Inverse of the above. The pilots that ship and stay shipped have, almost without exception:
That’s exactly what The Outcomes Method is engineered to produce — every one of those eight items maps to a specific phase gate or deliverable. The methodology exists because the alternative — hoping the pilot team gets all eight right by instinct — produces the failure rate the industry has been quietly tolerating for the last several years.
If you have a pilot in flight and you’re uncertain whether it’s tracking toward success or stall, the quickest diagnostic is to walk through the eight patterns above and ask, for each one: “have we actually done this, or have we assumed someone else has?” Most stalled pilots fail two or three of the eight; the recoverable ones fail one or two and can be corrected mid-flight.
If the answer is “we’re failing four or more,” the honest move is usually to pause, fix the engagement structure, and restart from the Design gate. That sounds expensive but is almost always cheaper than continuing on the current trajectory.
For pilots that haven’t yet started — where you’re still in the proposal stage — the AI Readiness Sprint is a structured way to surface all eight patterns before you commit. Two weeks, fixed price, written go/no-go at the end with explicit criteria for each of the eight.
If you want to walk through your specific situation first, the discovery call is 30 minutes, no agenda.
Related insights
AI Strategy & Roadmapping
What AU buyers actually pay for AI consulting in 2026 — who's pricing what, where the bands are moving, and how to read a proposal.
AI Strategy & Roadmapping
The numbers a CFO needs to see before approving AI investment — and how to keep them honest after the system ships.
Next step
30 minutes, no pitch, no deck — just a working conversation about how this applies to your situation.