AI testing lessons from a workflow that passed every test and still failed in production

At 9:14 on a Tuesday morning, I watched an AI workflow pass 200 test cases and then fail on a real support request from a hotel guest in Phoenix. The model classified the request correctly, the tool call returned 200, and the final answer still sent the guest to the wrong policy page. These are the AI testing lessons that matter: a passing test proves one path worked, not that the product is reliable in the conditions where people actually use it.

The failure was not a bad prompt. It was a system problem. The workflow had clean inputs, one model response, no retry, and a reviewer who already knew the expected answer. Real use brought stale data, an ambiguous request, a timeout, and a person who trusted the confident reply.

I've built two live multi-model AI platforms, and I keep seeing the same gap. Tests measure whether the workflow can complete a task. Production measures whether a person gets the right outcome while everything around the workflow is untidy.

A passing test is not the same as a passing workflow

The usual advice is to write more tests. That advice is incomplete. More tests built around the same clean assumptions only make you more confident in the wrong boundary.

A useful test for an AI workflow has at least four parts:

• The input a real person is likely to send
• The context and data available at that moment
• The actions the workflow can take
• The result the person actually needs

Most teams cover only the first and fourth parts. They ask whether the output contains the right answer. They do not ask whether the workflow found the correct record, used current data, handled a failed tool call, or made its uncertainty visible.

In my hotel example, the test said: "Find the cancellation policy for a reservation." The production request said: "I changed my flight and might arrive Thursday instead of tonight. Do I lose the room?" Those are not the same task. The second request contains a date change, a conditional question, a reservation lookup, and a policy decision.

The test passed because it removed ambiguity. The guest failed because the product had to manage it.

I now separate a response score from an outcome score. A response score asks whether the text is accurate. An outcome score asks whether the workflow moved the person toward a safe, correct resolution.

For an AI support workflow, outcome checks might include:

• Did it identify the right reservation?
• Did it use the policy version active on the booking date?
• Did it avoid promising an exception it could not authorize?
• Did it create a handoff when the answer depended on missing information?

This distinction changed how I review AI production failures. The model can be locally correct while the workflow is globally wrong. That is the first boundary you need to test.

The test set was too polite

Our test data had another problem: it behaved like a cooperative intern. Every field was present. Every date used the same format. Every document was readable. Nobody typed in all caps, pasted an old email chain, or asked two questions at once.

Real users are less considerate.

A customer may write, "Can I still cancel this? I already called yesterday and nobody got back to me." The workflow now has to identify the request, find the previous contact, determine whether the customer is referring to a different order, and decide whether a promise was made. A test that contains only "What is the cancellation policy?" says very little about real-world AI reliability.

The fix is not to invent random chaos. It is to collect failure-shaped examples from real use, strip private information, and add them to a living test set. I call this the Reality Set. It is not a benchmark for model quality. It is a record of the situations that make your product unreliable.

Build it from:

• Support tickets that needed correction
• Logs where users rephrased the same request
• Tool calls that timed out or returned empty results
• Cases where a human edited the model response
• Requests that caused a refund, escalation, or repeat contact

Then label the reason for failure. "Wrong answer" is too broad. Use labels such as stale source, missing field, ambiguous intent, bad retrieval, tool timeout, permission mismatch, or unsafe confidence.

At Zentoko, my multi-brand AI marketing platform, a request can look simple while crossing several boundaries. A user may ask for campaign copy, brand rules, audience context, an image brief, and an approval decision in one turn. A polished output can still be unusable if it violates the brand rules or skips the approval step.

The mistake I keep watching teams make is treating every failure as a prompt failure. Sometimes the prompt is fine. The input distribution is not.

State turns a good test into a bad customer experience

A test usually starts from a known state. The account exists. The document is current. The order is open. The user has not already tried three times. Production does not give you that courtesy.

This is where many AI workflows break. The model receives a partial view of a changing system and acts as if that view is complete.

Consider an internal IT workflow at a Verizon-sized company. An employee asks, "Can you add access to the finance dashboard?" The test environment may contain one directory record and one approval rule. Production may contain a transferred employee, two identities, an expired manager relationship, and an approval request already waiting in another queue.

The model can select the correct tool and still create the wrong result because the state was incomplete. That gap is quieter than a crash, which is what makes it dangerous.

I think of this as a State Contract. It defines what must be true before the workflow acts, what can change during execution, and what the workflow must verify afterward.

A practical State Contract answers these questions:

• Which record is the source of truth?
• How old can the data be before the workflow must refresh it?
• What happens if two records match?
• What does a successful write return?
• How does the workflow detect that another person changed the record first?

Do not let the model infer these rules from prose alone. Put checks around the model. Before a write, verify the record identifier, authorization, version, and current status. After a write, read the record back and compare the result with the requested action.

This is not glamorous engineering. It is the difference between "the API returned success" and "the user received what they asked for."

The same issue appears in memory. A long conversation can contain an outdated preference, a corrected address, or a decision that no longer applies. If your workflow treats every prior message as equally current, it is not remembering. It is accumulating risk.

Reliability fails at the seams

Most AI production failures happen between components, not inside the model response. The model says the right thing. The retriever returns the wrong document. The tool succeeds but the response parser drops a field. A retry repeats a refund request. A queue delivers the same job twice.

One workflow I reviewed used a model to extract invoice details, a Python service to validate them, and Postgres to store the result. Each component passed its own test. The production failure appeared between extraction and validation: the model returned a numeric value as a string with a currency symbol, the parser accepted it, and the validator treated an empty normalized value as zero.

The invoice was marked paid. The account was not.

That incident changed the way I draw test cases. I no longer test only functions. I test transitions.

For each handoff, check:

• What exact shape leaves the component?
• What shape does the next component expect?
• What happens when a field is missing, duplicated, or malformed?
• Can the operation run twice without causing a second side effect?
• Does a timeout leave the system in a known state?

Idempotency matters here. If a payment, email, ticket update, or database write can happen twice, give it an idempotency token and store the result. A retry should ask, "Did this action already happen?" It should not blindly do the action again.

The same rule applies to agent tools. A tool that returns an empty array is not necessarily a successful search. A model may read the empty result as proof that no record exists. Return explicit statuses: found, not found, ambiguous, unavailable, or unauthorized.

NIST's AI Risk Management Framework uses validity and reliability as separate concerns. That distinction is useful in practice. A system can be valid in a controlled test and unreliable when its dependencies, data, and timing change.

So test the seams. That is where the blast radius starts.

Observability has to explain the failure

A production alert that says "AI workflow failed" is not enough. It tells you that a customer had a bad experience, not why.

When I debug a workflow, I want one trace for the whole request. That trace should show the user input, retrieved records, model and prompt version, tool calls, latency, retries, validation decisions, final output, and any human edits. Store sensitive data carefully, but do not throw away the evidence needed to understand the failure.

A useful event record includes:

• A request ID that follows the workflow across services
• A workflow version and model version
• The source documents or record IDs used
• Tool latency, status, and retry count
• Validation results and blocked actions
• User correction, escalation, or repeat-contact signals

Without this trail, teams often blame the last visible component. The final answer looks wrong, so they rewrite the prompt. The real problem may be a stale index from six hours earlier or a timeout that caused the workflow to use a fallback record.

I also separate technical metrics from user metrics. Technical metrics include latency, error rate, token count, and tool success. User metrics include correction rate, repeat-contact rate, task completion, escalation, and reversal of an automated action.

A workflow that answers in 900 milliseconds but causes 12 percent of users to ask again is not fast in any meaningful product sense.

The 2025 METR study on experienced developers using AI tools found that developers took 19 percent longer on the studied tasks, even though they expected the tools to make them faster. That result is a useful warning: perceived success and measured outcome can move in opposite directions. Your team may feel productive while the system creates hidden review work.

Instrument that hidden work. Count the edits. Count the reopens. Count the "that is not what I asked" messages. Those signals are often more honest than a green test dashboard.

Put production evidence back into the test loop

A test suite that never changes is a museum. It may contain valuable artifacts, it just does not describe the current product.

Every serious failure should produce one of three outcomes: a code change, a workflow change, or a new test. Sometimes the right answer is to remove automation from that path entirely. That is not defeat. It is product judgment.

Create a weekly review with a small sample of real traces. Do not review only the worst failures. Review successful requests too, because a workflow can reach the right answer through an unsafe path. For example, it might return the correct refund amount while using an unauthorized account lookup. (And yes, that specific scenario has come up more than once in platforms I've worked on.)

A practical review cycle looks like this:

• Sample completed, corrected, abandoned, and escalated requests
• Classify the failure by data, state, tool, model, policy, or user experience
• Reproduce the request with the original versions and state
• Add the case to the Reality Set if it can happen again
• Ship a fix and rerun neighboring cases, not just the original case

Set release thresholds that match the action. A marketing draft can tolerate more variation than a payment reversal or a medical scheduling request. Do not use one quality score for every workflow.

For high-impact actions, require evidence before automation. The workflow should be able to show which record, policy, or calculation supported the decision. If it cannot, route the task to a person or ask the user for the missing fact.

At Ziggyloo, the children's AI tutor I built, the cost of a wrong answer is not just a low score. A child can learn the wrong method and repeat it. That means the product needs signals for confusion, repeated questions, and unexplained jumps in difficulty, not just a measure of whether the generated answer sounds fluent.

What I'm telling you is simple: the test suite should learn from the customer. If production cannot change your tests, you are collecting incidents, not improving the product.

What to do before your next launch

Start with one important workflow, not the whole application. Pick the path that can create financial, legal, operational, or trust damage when it fails.

Then run this review:

• Take 20 recent real requests and remove private data.
• Replay them with the exact model, tools, documents, and state versions.
• Add five cases with missing fields, stale data, duplicate records, and timeouts.
• Test every side effect twice and confirm the second attempt is safe.
• Review the trace for one successful request and one corrected request.
• Define the human handoff condition before you tune the prompt.

Do this before adding another agent, another model, or another layer of orchestration. More components increase the number of seams you have to verify. They do not remove the need for verification.

Here's the part most people miss: reliability is not a property you receive from a model. It is a property you build around data, state, tools, observability, and human judgment.

Your next test should not ask only, "Did the model answer correctly?" Ask, "Could a real person trust the outcome at 9:14 on a Tuesday, with stale data, a partial record, and one failed retry?" That is where the product lives.

FAQ

Why do AI workflows pass tests but fail in production?

Controlled tests use clean inputs, stable data, known state, and predictable tool behavior. Production adds ambiguity, stale records, timeouts, retries, duplicate requests, and users who interpret confident language as certainty.

What should I test in an AI workflow?

Test the full outcome, not only the generated text. Include real user inputs, retrieved data, state changes, tool failures, retries, authorization checks, side effects, and the human handoff path.

How do I measure real world AI reliability?

Track task completion, correction rate, repeat contact, escalation, reversed actions, tool errors, and stale-data incidents alongside model quality. A fluent answer with a wrong business outcome is still a failure.

How often should production failures become new tests?

Every repeatable failure should become a regression case after you remove private data. Review the test set weekly or at each meaningful workflow change so it reflects current users, tools, models, and source data.

Should every AI failure be fixed with a better prompt?

No. Prompt changes help when the model misunderstood a clear instruction. They do not fix stale data, bad identifiers, duplicate side effects, missing authorization, weak parsing, or an unsafe workflow boundary.