Turn Subject-Matter Experts into AI Workflow Designers

Give subject-matter experts a defined design task: explain the decisions inside a real case, identify the evidence each decision needs, and help test the proposed workflow. Pair them with someone who can turn that knowledge into software and someone who can approve the business rules. Their contribution should produce criteria another person can apply and a system the team can evaluate.

Give the expert a design mandate

The experienced coordinator, buyer, or team leader often knows which cases need extra care. Ask them to help define the workflow before an engineer builds it. Specify the supported task and the decisions you need to understand. “Tell us everything you know” creates an unbounded assignment with no clear completion condition.

Choose experts who do the work and people who receive its handoffs. A senior manager can explain the policy, while an experienced operator can explain how missing records affect a case. A newer colleague can reveal which instructions assume knowledge they don't have. Include the relevant perspectives without asking everyone to attend every session.

Assign complementary responsibilities. The expert explains the evidence and judges the proposed outcome. A workflow designer or engineer records the decisions and tests the implementation. The process owner approves the rule and scope. An expert can hold the owner role too, but state that authority explicitly rather than assuming experience grants it.

Protect time for the work. Arrange coverage for the expert's normal duties and schedule short sessions around concrete cases. Include review and testing time in the delivery plan. If you expect their judgement to shape the system, they need a way to reject an incorrect interpretation and see how the team resolves it.

NIST's AI RMF playbook asks organisations to establish the intended purpose, users, and context of an AI system. That supports beginning with the actual operation. It doesn't supply a ready-made rule for your company's exceptions or replace the process owner's decision.

A Reddit account of documenting team processes describes different explanations depending on who handles a case. Treat it as a reader's experience, rather than a prevalence estimate. The practical question is how your team turns those differences into a design everyone can inspect.

Replay cases to uncover judgement

Start with a recent completed case and its permitted records. Ask the expert to show what they saw at each decision, what they checked next, and why. Keep the sequence tied to the evidence available at that moment. A case's final outcome can make an earlier decision look obvious even when the operator lacked essential information.

Use prompts that expose cues. Ask what made them stop, which field they trusted, and what would have changed their decision. Ask which similar case required a different route. When they say a request looked unusual, ask them to point to the observable detail behind that judgement.

Capture off-system checks too. The expert may call a colleague, search old correspondence, or compare a request with a previous job. Record why they needed that check and which authorised source confirmed the answer. A proposed workflow that omits it may produce a confident output from incomplete evidence.

Sample cases deliberately. Include ordinary work, rework, an unresolved case, and an exception with a significant consequence. State that these examples help discover the rules; they don't establish the frequency of every branch. Use a separate, appropriately sampled review if you need a reliable volume estimate.

AHRQ's workflow interview guidance recommends a clear purpose, prepared questions, and a plan for collecting the information, including permission when recording. Its context is health IT. We apply the interview structure here to operational design, without assuming its examples prove outcomes in other sectors.

Explain how you will use the case material and restrict access to it. Use approved records or suitably designed synthetic examples, and keep identifying details out of notes when the design doesn't need them. Let the expert review your summary promptly while they can still recall the case.

If you need a full record of the current process, use our guide to mapping workflows and exceptions. Here, take the decisions inside that map and specify what the proposed workflow must do with them.

Turn explanations into decision records

Write a short record for each decision that affects the result. State the trigger, required evidence, rule, action, and next owner. Include the route when information is missing. Keep the record in plain language first so the expert can correct it before an engineer turns it into logic.

Distinguish recognition from authority. A model may recognise that a request mentions an urgent problem. That doesn't grant it authority to promise a response time or move another customer's booking. Specify what it can flag, what it can draft, and which person approves the consequential action.

Keep the evidence separate from the inference. A customer writes that a repair is covered; the current service agreement establishes whether the company covers it. Record which source decides and what happens when the source is unavailable. Don't let a persuasive message override the agreed evidence requirement.

Replace broad labels with operational descriptions. “Escalate complicated cases” gives an engineer little to implement and a reviewer little to assess. “Send a request with no confirmed service address to the coordinator, showing the candidate addresses and their source records” names an observable trigger and a concrete handoff.

Define what the receiver needs. Include the unresolved question, records already checked, and any deadline the owner has approved. A review queue that only contains the model's draft can force the receiver to repeat discovery. Test whether the handoff contains enough context to decide without opening several unrelated systems.

Record the rule's status and owner. An observation of current practice, a proposed improvement, and an approved rule describe different things. Keep that status visible so implementation work doesn't turn a draft into company policy. Link the rule to the cases that informed it and the person who approved it.

Resolve conflicting rules before implementation

Experts can disagree because they handle different work. A team serving contract customers may follow a rule that another branch never encounters. Start by comparing the case conditions and sources. If the difference reflects a valid boundary, put that boundary in the design and test both routes.

Other disagreements reveal a policy gap. An operator may offer a concession that another operator refuses, while the written policy says nothing. Give the process owner the evidence and the decision needed. Until they decide, route the case to an authorised person rather than making the model choose a new policy.

Separate an approved exception from a workaround. An expert may bypass a missing system field because it prevents routine work. Investigate why the field is missing and what control the workaround affects. The future design may require a data repair, a new field, or an explicit approval step.

Avoid settling disagreements by counting votes or picking the most senior speaker. Compare the sources, scope, and authority behind each answer. The owner may need specialist advice for a particular rule. Record the decision and its application date so staff and software use the same version.

Let people say that they don't know. A design session should expose uncertainty, not reward an immediate answer to every question. Give an unresolved item a named investigator and a completion condition. Keep it outside the automated scope when its resolution is necessary for safe operation.

For a PE portfolio, preserve valid local differences rather than treating the first company's practice as a group standard. For a corporate team, identify which department owns a disputed rule. Shared design can make decisions comparable while each authorised owner approves their own operating conditions.

Write acceptance criteria that another person can apply

Ask the expert what evidence would convince them that the result is correct. Convert that answer into a criterion with an input condition and an observable outcome. “Produces a high-quality draft” needs further definition. Name the fields, source requirements, and actions that determine acceptance.

Check the final state, not only the wording. A draft can read well while linking to the wrong account. A message can say that a case has entered review while the review queue has no record. Include checks on the receiving system and on any action the workflow must avoid.

Write criteria for uncertainty as carefully as for ordinary cases. If two records match and the evidence doesn't select between them, the workflow should preserve the ambiguity and ask for the missing information through the agreed route. A guessed answer should fail even if it happens to match the final case outcome.

Anthropic's January 2026 evaluation guide recommends unambiguous tasks and reference solutions, and describes domain experts contributing evaluation cases. It also distinguishes the agent's recorded interaction from the final state. Those ideas support writing criteria with operators and checking what the workflow actually changes.

Give a second knowledgeable reviewer the input and criteria without the first expert's explanation. Compare their verdicts and investigate disagreements. They may reveal missing context, an ambiguous rule, or a criterion that measures wording instead of the job. Fix the specification before using agreement as evidence of system quality.

Keep qualitative judgement where the task needs it. Some communications have several acceptable phrasings. A rubric can judge whether the draft states the known facts, avoids an unsupported commitment, and asks the necessary question. Test the rubric with people before relying on automated grading.

Design a service-request workflow together

Consider an illustrative maintenance company that wants AI to prepare draft job records from customer emails. This example uses invented conditions and makes no claim about a client outcome. A coordinator approves the draft before it enters scheduling; the assistant cannot promise a visit or decide a safety response.

The experienced coordinator shows a case where the customer's trading name matches two accounts. They used the site address and the current service agreement to choose the correct record. The design team records both cues and checks whether the new application can retrieve the authorised sources.

The process owner approves a rule: the draft must link to a confirmed account and service site, and ambiguous matches go to the coordinator. The engineer implements the candidate lookup and the review handoff. The expert writes examples where a name alone cannot establish the match.

Now replay a contrasting case. The email has the correct trading name but an outdated site address. The approved rule requires a confirmed site, so the assistant must flag the mismatch rather than select a record from the name alone. The expert's explanation becomes a negative test that checks an action the system must avoid.

Review the handoff with the coordinator who receives it. Can they see the original request, candidate records, and reason for escalation? Can they correct the draft through the approved interface? If they need to ask the expert to explain every case, improve the context or narrow the supported scope.

For a small company, the owner and expert may be the same person, with an external engineer building the workflow. For a startup selling to several customers, a forward deployed engineer can translate local cases into requirements for the shared product. In either arrangement, give the customer an explicit way to approve their rules.

Test outputs without coaching the system

Prepare a test set with agreed inputs and expected outcomes. Keep a separate group of cases out of prompt-tuning and implementation work where possible. Record which examples the team used while building. Performance on repeatedly discussed examples gives weaker evidence of how the workflow handles new cases.

Run the same conditions staff will use in the limited rollout. Include the permitted sources, access boundaries, and receiving-system checks. The expert should judge the output against the recorded criteria. Don't add missing facts verbally during the test unless the production workflow has an equivalent way to obtain them.

Keep the initial review independent of the builder's explanation. Show the reviewer what a real user receives, ask for their verdict, and then investigate the reason. If the engineer has to explain how to interpret every output, the interface or criteria need work. That explanation cannot substitute for a comprehensible operational result.

Classify failures by cause. A wrong source record needs different work from a missing business rule, an integration error, or a model that ignores the rule. Capture the input, relevant versions, expected outcome, and observed state. The team can then assign the fix instead of treating every failure as a prompt problem.

Repeat cases when model variation affects the result, and state the number of attempts. Track errors by case type as well as in aggregate. A small collection of easy cases cannot establish reliability for a rare exception. Our guide to building an AI evaluation set covers the technical testing process.

Before wider release, watch staff use the workflow under an agreed limited scope. Compare review time, corrections, and the final completed work with the baseline. Check whether the design moves hidden work to the receiver. An assistant that prepares drafts quickly may still create more investigation for the next person.

Keep the design current and share the knowledge

Give the approved decision records and test cases a home with clear owners. Store the rule version, approval, and supporting cases together. Link changes to the implementation release that applies them. Staff need to know when a rule changed and which workflow version they are using.

Reserve expert time after launch for reviewing new exceptions. A correction may expose a new case type, a changed source, or a disputed criterion. Assess that difference before changing the rule. Add the agreed case to the test set, then check existing cases before releasing the change.

Share design work with a second operator. Ask them to apply the records to an unfamiliar case and explain the evidence behind their decision. This tests whether the knowledge has become teachable and reveals where the documentation still depends on the original expert. It also creates a practical backup for review duties.

Keep participation credible by closing the feedback loop. Tell the expert which correction the team accepted, what changed, and how it performed in testing. Explain when the owner declines a proposed change. Our article on using staff corrections to improve AI describes how to maintain that loop.

Investors can inspect these records as evidence of delivery maturity. A startup that can turn customer judgement into maintained requirements and tests has more inspectable support for its deployment claims. A PE team can compare which cases transfer across companies and which need local approval, without treating copied documentation as proof of equivalent operations.

End the design phase with an observable handoff. Another operator can apply the criteria, the engineer can run the tests, and the process owner can explain the supported scope and unresolved items. Those are practical completion conditions for the expert's contribution. Continue funding their review time as the workflow and business rules change.