From pilot to production without starting again

Decisions to take in week one

None of these slow a pilot down materially, and all of them prevent a rebuild.

• Give the agent its own identity from the start, mapped to real identity-provider groups.
• Write the scope map as deny-all and grant deliberately, even when the pilot has one tool.
• Route inference through a mediated gateway so the model choice can change without touching agent code.
• Place approval gates on the actions that will need them in production, not the ones convenient in a demo.
• Turn on run recording and spend attribution on day one — retrospective data cannot be created.
• Use real data where lawful, not synthetic data that hides edge cases.

Scope the pilot narrowly, build it properly

The useful discipline is a narrow task with production-grade plumbing: one process, one team, real data where permitted, and the full control path in place. That produces something a security reviewer can assess and an operations team can adopt.

Broad pilots with placeholder governance produce impressive demonstrations and no route forward. They look good in slide decks but collapse under production questions.

Narrow scoping has another advantage: it lets you measure. A pilot that handles one specific case type end-to-end will teach you more about latency, quality, cost and failure modes than a broad pilot that touches many processes superficially.

Metrics that matter

Pilot success should be measured against production criteria, not demo criteria. The most useful metrics fall into four groups.

• Quality — accuracy, consistency, and the rate at which human reviewers disagree with the agent.
• Governance — policy coverage, approval coverage, identity mapping and record completeness.
• Cost — spend per transaction, per case and per model, including internal infrastructure.
• Operational — latency, availability, error rates, time to detect failure, time to recover.

Production readiness checklist

Before promoting a pilot, the organisation should be able to answer yes to a consistent set of questions. The exact list varies by sector, but the shape is always similar.

• Can we name every identity that may act on behalf of the agent?
• Can we reproduce any decision from the last six months using only records we hold?
• Can we revoke the agent's access to every system in under an hour?
• Can we switch the primary model without changing application code?
• Do we know the all-in cost per transaction at current volume?
• Have we tested the incident response path with a realistic scenario?

Organisational readiness

Technology is only part of the production problem. The agent needs an owner, an operations model, an escalation path, a training plan for users, and a change-management process for policy updates.

The owner should be a named role, not a shared team mailbox. They need authority over scope, model selection and approvals, and they need to be accountable when things go wrong.

Users need to understand what the agent is for and what it is not for. Over-reliance on an automation that has not been trained into the organisation creates second-order risks: people stop questioning outputs, exceptions are not escalated, and the system silently degrades.

What good looks like at the end

A finished pilot should leave you with a working process, a policy you can extend to the next one, evidence a reviewer has already seen, and a cost figure you trust. Promotion then becomes a decision rather than a programme.

It should also leave you with a template. The next agent should reuse the identity model, the policy patterns, the approval flow and the observability structure. Each subsequent deployment becomes cheaper and more predictable.

The organisations that scale agents successfully are not the ones with the most impressive pilots. They are the ones whose pilots were built to be promoted from the first day.