Most AI pilots fail not because the technology did not work, but because the pilot was never designed to answer a question. It was designed to impress someone.
MIT’s Project NANDA study of over 300 generative AI initiatives found most showed no measurable six-month return. Gartner expects over 40% of agentic AI projects to be cancelled by end of 2027.
A pilot done properly is the cheapest insurance against joining those numbers. Here is the structure.
What a pilot is for
Worth stating plainly, because the word covers two different activities that get confused.
A pilot answers a question. Will this work on our data, at our volume, at a cost that repays? It ends in a number and a decision, including the decision not to proceed.
A proof of concept demonstrates a possibility. It shows something can be done. It runs on selected examples, skips error handling, and exists to persuade.
Both are legitimate. They are not the same purchase, and buying one while expecting the other is a reliable route to disappointment.
The test: could this end in “no”? If the engagement cannot produce a negative answer, it is a demonstration rather than a pilot, and the decision was made before it started.
Before you start: write the decision down
A pilot exists to produce a decision, not a demonstration.
Before anything is built, write one sentence:
“We will build this if the agent handles at least X% of cases correctly with an exception rate below Y%, at a run cost under £Z per month.”
Fill in the numbers. If you cannot, you are not ready — because you have no way to tell success from failure, and the pilot will end in opinion rather than evidence.
Three numbers you need:
- Accuracy threshold. What does it have to get right? Be realistic. 100% is not available from any system, including the humans doing it now.
- Exception rate ceiling. What proportion can be escalated before the saving disappears? If 40% still need a human, the labour saving is modest.
- Run cost ceiling. What monthly model cost makes this uneconomic? Model this before building.
Establish the baseline first
You cannot prove improvement without knowing the starting point, and almost nobody measures it.
For one week, measure the existing process:
| Measure | Why |
|---|---|
| Time per task | The labour saving comes from here |
| Tasks per week | Volume determines whether a build repays |
| Current error rate | Critical. Humans make errors too. |
| Cost of an error | What a mistake actually costs to fix |
| Fully loaded hourly cost | Salary plus employment costs, not salary |
The current error rate is the one people skip, and it changes the whole conversation. If your team already gets 6% wrong, an agent at 94% is performing at human level — and holding it to 99% is a standard you do not apply to the people doing it now.
The five stages
Stage 1 — Feasibility on real data (2–3 weeks, £6,000–£12,000)
Not a demo on sample data. Run the hardest version of the actual task, on your actual records, including the messy ones.
What you get: a measured accuracy number, a list of where it fails, a per-task cost, and a go/no-go.
What you must not accept: a polished demonstration on curated examples. That tells you nothing about production, where the inputs are not curated.
Deliberately include the awkward cases. A pilot that only sees clean data is testing the wrong thing. Give it the invoice that arrived as a photograph of a screen.
If it fails here, you have spent £6,000–£12,000 instead of a build budget. That is the pilot working, not failing.
Stage 2 — Narrow build (3–5 weeks)
One process, properly engineered. Error handling, logging, a defined escalation path.
Resist expanding scope. “While we’re at it” is how a pilot becomes a project that has to succeed. Narrow means you can still walk away.
Stage 3 — Shadow running (2–4 weeks)
The step nearly everyone skips, and the most valuable one.
The agent runs alongside the existing process with no authority to act. It processes the same work your team processes. Nobody is affected by its output.
Then compare. Where they disagree, investigate.
What this surfaces:
- Edge cases nobody thought of
- Cases where the agent is right and your current process is wrong (this happens more than people expect)
- The real exception rate, on real volume
- The real run cost, on real volume
Shadow running is how you find out whether it works without anything being at risk. It costs a few weeks and it is the difference between a controlled launch and a public failure.
Stage 4 — Limited authority (2–4 weeks)
The agent acts, but narrowly:
- Only the highest-confidence cases. Everything else escalates.
- Only a proportion of volume — start at 20%.
- Only reversible actions, where possible.
- With a human spot-checking a sample daily.
Expand the boundaries weekly as evidence accumulates. Pull them back immediately if anything surprises you.
Stage 5 — The actual decision
Against the numbers you wrote at the start:
- Did it hit the accuracy threshold?
- Is the exception rate below the ceiling?
- Is the run cost under the limit?
- Does the payback period work?
Then decide — including the option to stop. A pilot that concludes “no” has done its job and saved you a build.
Choosing what to pilot
The commonest mistake is piloting the most interesting process rather than the one with the strongest arithmetic.
Score your candidates against six characteristics:
- High volume — hundreds of occurrences a month
- Unstructured input — documents, emails, varied formats
- Judgement, not expertise — the kind a trained person makes in seconds
- A checkable right answer — a PO to match, a rule to apply
- Recoverable errors — nothing irreversible
- Currently consuming measurable hours
Fewer than four: be sceptical. All six: run a feasibility sprint.
Then pick the one with the highest annual cost, not the one that would demonstrate best. A pilot exists to produce a decision, and the decision that matters is about the process costing you most.
Avoid piloting anything customer-facing first. An internal process gives you room to be wrong. A pilot that misfires in front of customers ends the programme regardless of what the numbers said.
What makes a pilot worthless
It ran on curated data. Production data is messy. A pilot on clean examples tells you nothing.
There was no baseline. Without the current error rate and time per task, “it seems to work well” is the only available conclusion.
Success was never defined. The pilot ends and everyone argues about whether it worked.
Scope grew during it. Now it has to succeed, because too much has been spent.
It was a demo. Built to persuade, not to survive. These do not become production systems by extension — they get rebuilt.
Nobody measured the run cost. The thing works and is uneconomic, which you discover in month three.
How long and how much
| Stage | Time | Cost |
|---|---|---|
| Baseline measurement | 1 week | Your time only |
| Feasibility on real data | 2–3 weeks | £6,000 – £12,000 |
| Narrow build | 3–5 weeks | £12,000 – £30,000 |
| Shadow running | 2–4 weeks | Minimal — monitoring |
| Limited authority | 2–4 weeks | Minimal |
| Total to a defensible decision | 10–17 weeks | £18,000 – £42,000 |
You can stop after feasibility for £6,000–£12,000. That is the point of structuring it this way — the expensive commitment comes after the evidence, not before.
Who needs to be involved
A named owner. One person accountable for the outcome. Without this, the project degrades after launch.
The people who do the work now. They know the edge cases and they will spot where the agent is subtly wrong. They are also the people whose jobs change, so involving them early is both practical and decent.
Someone who can say stop. A pilot nobody can cancel is not a pilot.
Managing the people side
The part that derails more pilots than the technology does.
Tell people what is happening, early and plainly. A pilot discovered rather than announced is read as a redundancy exercise, and the people whose cooperation you need become the people quietly proving it does not work.
Be straight about what it means for jobs. In most cases an agent removes part of a role — the repetitive, judgement-light part — rather than a whole one. Someone still handles exceptions, and someone must own the system. If headcount genuinely is the objective, say so; people work that out regardless, and discovering it late is worse.
Involve the people who do the work in the design. They know the edge cases, and they will spot where the agent is subtly wrong far faster than a developer will. They are also your best source of the awkward real-world examples feasibility needs.
Do not let the sponsor be the only voice. Projects commissioned at board level and designed without the operators consistently produce systems that handle the process as described rather than as performed.
Agree in advance what happens to freed capacity. “We will use the time better” is not a plan, and it is the question everyone is actually asking.
The contract for a pilot
A pilot is a small engagement and it still needs the basics written down, because pilots that succeed become projects and the terms carry over.
Fixed price. A pilot on time-and-materials has no boundary, which defeats the point of a pilot.
A defined deliverable, not a time period. “A measured accuracy figure on your real data, a cost-per-task model, a list of failure modes and a written recommendation” — not “three weeks of work”.
Your data stays yours, and you are told in writing where it is processed and by whom before anything is sent. The data protection questions are here.
You keep the evaluation suite. The test set and the measurement harness are the most durable thing a pilot produces. If you proceed with another supplier, that work should go with you.
No obligation to continue. A pilot that commits you to a build is a sales process with a technical component.
The report is yours, including the right to take it to another firm.
What the pilot should produce
Six artefacts. If a pilot ends without these, it has produced an impression rather than a decision.
- A measured accuracy figure on real data, including the awkward cases
- A measured exception rate at realistic volume
- A measured cost per task, extrapolated to monthly volume
- A list of failure modes — what it got wrong and why
- A baseline comparison — how the current human process performs on the same measures
- A written go/no-go against the criteria agreed before the pilot started
The baseline is the one most often missing, and it changes conclusions more than anything else. Teams that have never measured their own error rate are routinely surprised, and an agent judged a failure at 93% can look very different against a human baseline of 94%.
After the pilot succeeds
A pilot that passes is not a system. Four things stand between them, and budgeting for the pilot alone is a common planning error.
Hardening. Error handling for every failure mode shadow running surfaced, retry logic, graceful degradation when a dependency is unavailable. Typically two to four weeks.
Governance. Audit logging, authority boundaries enforced in code, approval workflows, a kill switch a non-technical person can operate. Cheap now, impossible to retrofit after an incident.
Integration at production scale. A pilot reading a few hundred records differs from a system reading thousands daily. Rate limits, batching and failure handling all become real.
Monitoring. Exception rate, cost per task, latency, with alerting. Someone must notice degradation before the quarterly review does.
Budget roughly the pilot cost again to get from a passing pilot to a production system. A proposal that treats the pilot as most of the work is describing a prototype.
Three pilots that should have stopped earlier
Patterns worth recognising, because each wasted months that a clear criterion would have saved.
The one with no baseline. Accuracy came in at 91%. The sponsor judged it inadequate. Nobody had measured the existing human error rate, which later turned out to be comparable. The pilot had succeeded and was recorded as a failure.
The one that kept expanding. Feasibility passed on invoices. Someone asked whether it could handle purchase orders too. Then delivery notes. Each addition was reasonable; collectively they turned a decision into a project that had to succeed, and walking away stopped being possible.
The one that proved the data was the problem. Accuracy at 58%, all failures traceable to contradictory source records. The correct response was to stop and fix the data as its own project. The actual response was four more weeks of prompt adjustment against inputs that could not support a better answer.
The common thread: no criterion agreed in advance. Without one, a pilot cannot conclude — it can only continue until someone loses patience.
Frequently asked questions
How long should an AI pilot take?
Ten to seventeen weeks from baseline measurement to a defensible decision. Feasibility alone takes two to three weeks and can end the process cheaply if the answer is no.
What should an AI pilot cost?
£6,000–£12,000 for feasibility on real data. £18,000–£42,000 through to a full go/no-go including a narrow build and shadow running. Stopping after feasibility is a legitimate and inexpensive outcome.
What is shadow running?
Running the agent alongside your existing process with no authority to act, comparing outputs and investigating disagreements. It surfaces real edge cases, the real exception rate and the real run cost with nothing at risk. It is the most valuable stage and the most commonly skipped.
What accuracy should I require?
Start from your current human error rate — most organisations have never measured it and are surprised. Requiring 99% from an agent when your team achieves 94% is a standard you do not apply internally. What matters more is whether errors are caught by escalation.
How do I know if the pilot succeeded?
By comparing against numbers written before it started: accuracy threshold, exception rate ceiling, run cost limit. Pilots without pre-agreed criteria end in opinion.
Should I pilot with a vendor’s demo?
No. Demos run on curated data and prove nothing about your messy reality. Insist on feasibility against your own records, including the awkward ones.
What if the pilot fails?
Then it worked. You spent £6,000–£12,000 to avoid a build that would not have paid off. That is the cheapest possible version of that discovery.
Written by Jas Singh, Founder. Last updated 8 October 2026.
Figures from MIT Project NANDA (August 2025) and Gartner’s forecast on agentic AI project cancellation, reported via secondary sources and cited as directional.
