Why an AI Pilot Works in One Team and Fails in Another

An AI pilot gives evidence about a workflow under particular conditions. When another team gets a different result, compare those conditions before changing the model or blaming adoption. Start with the cases people handle, the information they can access, and the decisions they can make. Then test whether the receiving team can complete acceptable work with the same design.

Recover the conditions behind the first result

Ask what the pilot actually completed. A team may report faster ticket triage because an assistant proposes the right queue for routine requests. That result doesn't establish that the assistant can interpret a disputed service contract, resolve a customer complaint, or authorise a repair. Write the task boundary and the accepted outcome before discussing wider use.

Recover the first team's operating record. Include the input channel, permitted documents, model and application versions, and review process. Record the cases the pilot excluded. Find out whether its members had dedicated support, cleaned incoming records manually, or asked an engineer to fix exceptions during the test.

Those activities can belong in a sensible first pilot. They also consume effort that the receiving team needs to provide or remove through a tested change. Count preparation and exception work alongside the time spent in the application. A workflow that depends on its builder sitting beside users needs a plan for operation after that support ends.

In its April 2026 discussion of the jagged technological frontier study, Harvard Business School explains how AI helped on some consulting tasks and hurt performance on a different task. The underlying experiment used an earlier generation of GPT-4. It supports checking the actual task boundary; it doesn't establish the capability of your current model or predict your rollout's result.

Use a concrete transfer brief. In the illustrative example throughout this article, a service company's North team handles mostly standard warranty requests. Its South team handles more multi-supplier contracts and disputed coverage. Both want help proposing a ticket queue, with staff keeping routing approval. The same department label conceals different work.

Compare the work before copying the workflow

Walk through recent cases with the receiving team. Start at intake and follow each case to an accepted route, including the point where staff need more information. Ask what makes the case straightforward, what causes rework, and who resolves an exception. Compare this with observed work in the first team.

Distinguish required local differences from avoidable variation. A contract can require a different approval path. An old spreadsheet used because staff never received access to the service system may be a process defect to fix. Record the reason for each difference and ask the process owner to decide which rules should change.

Check the unit of work. North may count each incoming email as a ticket, while South groups a chain of emails into a customer case. Their tickets-per-hour figures then describe different denominators. Agree how to count intake, duplicates, completed cases, and reopened work before comparing results.

Don't force the receiving team to match a presentation diagram. Use the map to identify what the application assumes. If the integration assumes every request has an asset identifier, inspect how South handles requests without it. That is a requirement to resolve through intake design or exception handling.

Our guide to mapping a real workflow and its exceptions gives a method for collecting that evidence. Keep the transfer review focused on differences that change scope, correctness, or the team's ability to operate the result.

Test local data and access

A shared system name doesn't guarantee shared meaning. The two service teams may use the same ticket platform while entering contract identifiers differently. A field called priority may describe a contractual response commitment in one team and a supervisor's preference in the other. Validate the meaning with the person accountable for the source.

Trace the information required for a routing decision. For South, that may include the current service agreement, asset ownership, and the supplier responsible for the repair. Check which source is authoritative, how staff know it is current, and what the workflow does when sources disagree.

Test retrieval through the receiving user's approved access. A pilot account with broader permissions can produce a draft that ordinary staff cannot verify. Avoid expanding access merely to reproduce the pilot's score. Restrict the task to permitted information and route missing context to the appropriate owner.

Inspect the full input path. A tidy exported contract may reach the pilot correctly while South's live requests arrive as forwarded messages with scanned attachments. Test the conversion and record matching, not only a model prompt pasted with the right context. Keep confidential examples within the company's permitted environment.

Build a local evaluation set containing ordinary cases and exceptions that change the correct route. Have authorised reviewers agree the expected decision or acceptable alternatives. Include cases where the correct behaviour is to ask for information or decline to propose a route. Preserve a held-out set so changes don't pass solely because the builder tuned against every example.

A Reddit discussion about pilots reaching production asks what blocks wider use and raises concerns about data and continuing ownership. Those comments provide reader questions rather than a reliable failure rate. Test the local data path and support arrangement directly instead of adopting the thread's numerical claims.

Check experience, incentives, and decision rights

Observe how staff review the same proposal. An experienced warranty clerk may recognise a wrong supplier immediately, while a new employee needs to open several records. The application's first-draft speed can stay constant while the team's review effort changes substantially. Train and measure the full task under the receiving team's conditions.

Brynjolfsson, Li, and Raymond's 2023 working paper on generative AI in customer support reports different productivity effects across workers, with larger gains among less experienced and lower-skilled agents in that setting. That finding doesn't imply that a novice can safely review every AI output. Assess whether local reviewers can detect consequential errors in the particular workflow.

Check what the team gets credit for. If managers reward rapid ticket closure, staff may accept a proposed queue to clear their own list even when it creates downstream rework. Measure correct routing and reopened cases with the receiving queue. Give staff a way to reject an inappropriate proposal without making their performance figures look worse.

Ask who can decide when a request crosses departments. North's supervisor may control the whole warranty queue. South may need a contract manager's approval before assigning a supplier. Give the assistant an explicit pending state and escalation route when staff lack authority. Software should preserve that decision boundary.

Protect time for training and feedback during ordinary operating hours. A volunteer pilot group may have shaped the interface over several sessions; a receiving team shown a brief demonstration has had a different preparation. Plan training around real cases and check that each role understands the exception route.

Review the manager's workload too. If every uncertain draft now needs a supervisor who already handles escalations, the workflow concentrates demand at an existing constraint. Our guide to rolling out AI when managers are overloaded covers how to allocate that time and remove competing work.

Recalculate review demand at local volume

Use arrival patterns, not only a daily average. A small pilot can handle exceptions as they appear, while a receiving team gets a surge after a weekend or a large customer incident. Record the cases arriving in those periods and the staff actually available to review them.

For an illustrative capacity calculation, suppose a team receives 24 cases an hour, a quarter need specialist review, and each such review takes eight minutes. That creates 48 specialist-minutes of demand per hour before ordinary duties, interruptions, or unusual cases. These figures are planning assumptions, not a staffing benchmark or a measured result from a client.

Even an average below available capacity doesn't establish an acceptable waiting time. Cases arrive unevenly and some take much longer. Test the peak pattern with the people who handle it. A specialist who has other work cannot provide a full hour of review in every clock hour.

Count all required checking. If routine proposals also need human confirmation, include that effort separately. Inspect the long tail of cases that need repeated review or another department. Reducing first-response time while increasing unresolved work doesn't demonstrate a better completed process.

Set an admission limit for the supported cohort and keep the manual process available. Define what happens when the review queue exceeds its agreed capacity, including which work receives priority and who can pause assisted intake. Confirm that the fallback team has actual cover.

Volume can also expose technical limits. Test the integration's queueing behaviour, timeouts, and downstream service limits under representative demand. Keep software latency and human waiting time visible separately so the team can act on the right constraint.

Separate case mix from a performance change

Break the aggregate result into meaningful case groups. Compare routine requests with routine requests and contract exceptions with contract exceptions. Add local categories where they change the task or acceptance criteria. Report the number of cases in each group so a very small sample doesn't look conclusive.

Consider an illustrative calculation using proposals accepted without correction. Assume the rate is 90% for routine cases and 50% for exceptions in both teams. In a sample with 80 routine cases and 20 exceptions, staff accept 72 plus 10 proposals, or 82 of 100. With 40 routine cases and 60 exceptions, they accept 36 plus 30, or 66 of 100.

The aggregate falls from 82% to 66% even though the assumed rate within each group hasn't changed. That arithmetic shows how a different case mix can change a headline result. The figures are constructed to explain the comparison; they don't describe an acceptable quality threshold, and acceptance without correction doesn't prove correctness.

Have qualified reviewers assess completed cases against the agreed business outcome. Inspect wrong routes, delayed escalations, and rework after approval. Keep consequential failures visible even if an average improves. A favourable aggregate must not hide an access breach or an unauthorised action.

Separate scope coverage from performance inside scope. A team can raise its acceptance rate by sending more difficult work to the manual queue. Report how much eligible work entered the assisted path and what happened to the rest. Track the combined workflow so that exclusions don't make the operating result disappear.

Use a local baseline and a contemporaneous comparison where practical. Account for staffing, seasonal demand, and policy changes. If the design and permissions allow a controlled comparison, assign comparable cases or rollout groups in advance. Otherwise label the limits of a before-and-after result and avoid claiming that AI caused every change.

Run a bounded transfer test

Write a transfer record that names the reusable components and the assumptions to test locally. The ticket connector and review interface may stay common. South's contract mapping, permitted sources, and escalation ownership need local review. Keep a versioned record of these choices so the next deployment doesn't inherit an undocumented patch.

Start by replaying approved historical cases or running in shadow mode, with the existing process controlling real routing. Inspect errors with local operators and correct the input mapping or task boundary where the evidence supports it. Shadow evaluation cannot show how staff behave when they depend on the tool, so treat it as a preparation stage.

Move to a limited supervised cohort after the required owners accept the scope and controls. Define eligible cases, reviewer cover, and a review date. Keep a record of corrections and the time people spend making them. Ask the receiving queue to confirm the eventual outcome.

Demonstrate the pause and recovery path with the receiving team. Include an unavailable contract source, a slow downstream system, and a case awaiting an absent approver. Keep uncertain actions inspectable rather than retrying them blindly. Our guide to shipping an AI workflow with a safe fallback covers that release boundary.

Track changes made during the transfer test. If a new model, revised contract source, and redesigned review screen arrive together, the result won't identify which change helped. Where practical, evaluate a specific change against fixed cases and then confirm its effect in the operating workflow.

Ask a local maintainer to handle a test failure before wider release. The first team's engineer can help, but ongoing operation needs a receiver with access, instructions, and support cover. A transfer that requires permanent attention from the pilot's creator needs an explicit continuing arrangement.

Decide what to standardise and what to change

Choose among expanding the tested scope, adapting a defined part, or stopping that design for the receiving team. Record the evidence and the owner of the decision. A local difference may justify a narrower assisted task without invalidating the first team's result.

Standardise components whose behaviour and maintenance the receiving team can accept. That can include a connector, audit record, review control, or evaluation harness. Give each common component a maintainer and a version. Keep local business configuration separate so that a change to one company's contract rule doesn't alter another team's deployment.

Require evidence for local exceptions too. Ask which operating condition needs the difference and who approves it. If several teams need the same change, review whether it belongs in the shared product. If a local workaround only compensates for a broken source process, assign the underlying repair.

For PE operating teams, include company-level authority and data boundaries in the transfer decision. For a software startup, examine which customer-specific work feeds the product roadmap and which needs a scoped service. Our guide to moving customer-specific FDE work into product examines that choice.

Keep the comparison record for the next rollout: supported case types, source requirements, review demand, and unresolved limits. Update it as live conditions change. The next team can then test named assumptions instead of trying to reproduce a headline productivity figure.

Before approving expansion, have the receiving owner walk through an ordinary case and a difficult exception using their own permitted records and staff. Inspect the accepted result and the work required to reach it. That demonstration gives the rollout decision a local operating basis.