Two approaches to diagnose an AI agent's confident wrong answer
Shraddha and Harshal's postmortem for in-product agents before you change the model
I co-wrote this with Shraddha Bodhe. She builds agents that help engineers investigate yield problems in semiconductor manufacturing, and she writes about that work on AI from the Trenches.
I build AI agents into software products. Comparing notes with her made it clearer which failures we share, and where the cost of a mistake changes what we should expect from an agent.
We brought both perspectives into this article. The examples are generalized from our work. The shared question is why an agent can give a convincing answer and still get the job wrong.
This version is the cross-post of our joint piece. Read her publication of it on Before You Blame the Model.
First published with Shraddha Bodhe on Before You Blame the Model (AI from the Trenches), 2026-10-22.
Related:
Diagnose before you upgrade
When an agent gets something wrong, changing the model is an appealing first move. A stronger model might follow instructions better, reason through a hard problem, or make fewer mistakes. Teams can also make that change quickly, often faster than investigating the failure.
The hard part is knowing whether the upgrade addresses what went wrong. If the agent never received the error message, a stronger model still works without it. If the team has not agreed on what counts as success, comparing two models will not settle that disagreement. If nobody recorded the relevant data or changes, the team has little basis for choosing a fix.
We have run into these problems in software products and in semiconductor manufacturing. The consequences differ. A user can often inspect and correct a draft workflow. An engineer acting on a bad manufacturing recommendation may not discover the mistake until a batch has gone through several more production steps. In both settings, we need to understand the failure before deciding whether the model is the part to change.
Three questions have been useful for that diagnosis:
What did the agent know before it acted?
What was correct supposed to look like?
What can we reconstruct after the fact?
What did the agent know before it acted?
I first tried model upgrades while building BrainOS, an AI second brain for my own work. I wanted the agent to reach the decisions I would make. I built versions of this for My Senior PM Job Search System and for product management. When it gave a different answer, I told it what I thought was right. It agreed and revised the response, then made similar mistakes later.
I moved to stronger models. The reasoning improved. The cost of each question also rose. Some disagreements remained. I then started asking what information I had used that the agent was missing.
There were gaps in its Slack and document context. Searching Slack for every question was expensive, so I built daily jobs to gather meeting transcripts, task-board updates, relevant messages, and other work records into the workspace. The jobs combined deterministic steps with a local or inexpensive model. That gave the agent more of the background I was using to make decisions.
The model upgrades improved one part of the system. The missing information needed a separate fix.
In a workflow copilot my team built, the gap was more immediate. A user could ask, "Can you fix this?" The agent might receive the chat text and part of the workflow graph, but not the exact error, the sample input that broke the run, or the status of the Slack or Gmail integration it depended on. It could suggest a fix that left the workflow failing in the same way.
We use session context to describe information about what the user is doing at that moment: the open workflow, the selected step, the failed input, and the actions available to the agent. Users reasonably expect a copilot inside a product to have access to much of what they can see. Without it, even "fix this" becomes ambiguous. These choices support the day one expectations users bring to AI copilots.
Passing more information indiscriminately did not solve the problem. We initially added relevant files and best practices until the copilot had hundreds of sentences of guidance, much of it unrelated to the immediate request. We moved to retrieving relevant information for each prompt. A failed workflow run needed its error details. Other requests needed different context.
That required product decisions as well as engineering work. We had to choose which parts of the session the copilot could access, how users indicated their intent, and which actions the agent could take.
In semiconductor manufacturing, Shraddha sees a different missing piece. The agent may have the test results without knowing what changed in the way those results were produced. The following example is hypothetical. It shows the kind of context an engineer would need.
On Monday morning, an engineer sees that the percentage of chips passing final testing has fallen from 94% to 88%. The agent finds that the affected batches went through a machine serviced on Friday. It produces a timeline, highlights the maintenance event, and recommends investigating whether the machine was returned to production too soon.
There is a reasonable lead and a chart to support it. Friday also brought a new version of the software that tests the chips. One customer now requires a tighter limit on an electrical measurement. Some chips that would have passed on Thursday fail on Monday, even if their measured behavior has not changed.
The engineer who approved that release knows to check whether the drop reflects a change in the chips, a change in the acceptance criteria, or both. The agent received the pass/fail results and maintenance history, but not the test-program release note. It has no basis for treating Thursday's 94% and Monday's 88% as comparable. Tracking test-program revisions and changes to limits matters because the meaning of a pass can change between versions.
We call the knowledge behind that comparison institutional context. Here, it includes why the limit changed, which customer it applies to, and how to connect a test result to the criteria used to judge it. The numbers alone do not tell that story.
A useful next step would be to compare the actual measurements under consistent criteria and check which failures the new limit accounts for. That would not rule out a manufacturing problem. It would stop the team from treating a lower pass rate as sufficient evidence against the machine. Changing models before supplying that context would leave the central gap unresolved.
Both kinds of context matter in both settings. A workflow copilot needs the current error and the team's conventions. A manufacturing agent needs the batch's current status and the background behind a production decision. When investigating a failure, check exactly what reached the model, including information that was available somewhere in the organization but absent from that request.
The MAST study, based on more than 1,600 annotated traces across seven multi-agent frameworks, identifies missing conversation history and disobeyed specifications among its failure modes, alongside coordination and verification problems. It offers several explanations to investigate before attributing a failed task to model capability.
What was correct supposed to look like?
Even with the right information, an agent can disappoint because the team and the user expect different things from it. A model comparison needs a clear definition of a better result.
In community research for one workflow product, I found that users valued an agent that got them 70–80% of the way to a finished workflow. They could inspect the draft, correct it, and complete the remaining work. I saw similar behavior in coding use cases.
That observation describes useful progress. It is not an acceptable error rate for every task. A draft can save most of the setup work and still need review before it runs. Those users understood the work well enough to recognize mistakes, and correcting them was relatively cheap.
In manufacturing, an engineer may act on a recommendation and send a batch to the next production step. By the time a mistake becomes apparent, reversing the decision may no longer be possible. The same amount of unfinished work or uncertainty can have very different consequences.
Return to the falling pass rate. Suppose the agent now has the release note and correctly identifies the tighter test limit. The engineer's next question is, "Which batches need review before today's shipment?"
An explanation of the six-percentage-point drop does not answer that question. The engineer needs to know which batches were tested under the old limit, which are intended for the customer with the new requirement, and whether their recorded measurements are sufficient to assess them against it. A batch that passed the old test cannot automatically be treated as meeting the new requirement.
The agent could correctly explain the dashboard and still give an incomplete list of affected batches. It could also flag every batch of that product, leaving the engineer to work out which ones actually need attention. Those answers might look equally cautious in a generic evaluation. They create different problems. One could miss a shipment that needs review. The other could unnecessarily delay unaffected orders.
For this task, we would define success as identifying the batches that need review, showing why each was included, and marking any whose customer assignment or test history is missing. The engineer would make the release decision. We would evaluate the agent against cases with known affected and unaffected batches, including incomplete records, rather than score only how convincingly it explained the yield drop.
My work showed another way to misdiagnose an agent problem. In one case, a user asked a question, and the agent both diagnosed the issue and changed the product. It explained none of the changes and offered no undo. Even if the edits were correct, the agent had done something the user did not expect.
My team made the agent show what it changed and let users choose between diagnosis and action. For changes that are difficult to judge, a preview or explanation gives the user something specific to review. Ambiguous requests may need confirmation. Applied changes need a recovery path where one is possible.
This is why I treat accuracy, usability, and observability as related requirements. Checking whether an edit is correct does not establish whether the user wanted it made.
The agreement also affects how an agent should handle delegated work. Someone asking a quick question wants an immediate response. Someone handing over a workflow build may accept a longer wait for completed work. Repeated interruptions for routine approvals undermine that delegation. The product needs to establish what the agent can do independently and when it should return to the user. For delegated work, I would take a slower agent with reliable recovery over a five-second answer I could not reverse.
Gartner's guidance on agent governance distinguishes agents that observe, advise, act with approval, and act autonomously. It also recognizes that advice can influence consequential decisions even when a person takes the action. Human review is useful only when the reviewer has enough information and time to assess the recommendation.
Before blaming the model for a poor result, establish whether it failed an agreed task, used the wrong definition, or acted beyond what the user intended. Those problems call for different changes. An accuracy score can obscure the distinction.
What can we reconstruct after the fact?
The first two questions are hard to answer if the team cannot see what happened. A prompt, response, and list of tool calls may leave out the evidence needed to explain a failure.
When my team investigated a bad answer, we also needed to know which workflow the user had open, what was selected, and what the agent changed. We began capturing a before-and-after record for the same action: the relevant user-visible state, a structured version we could compare, and the tool or API calls with their inputs and outputs. AI observability needs to capture this user-visible state. Those records helped us check whether the agent had received the wrong context, made a bad decision with the right context, or applied an unintended change.
We started with one common job rather than recording every possible interaction. When we needed to see a session in motion, we linked to an existing session replay. A product-specific viewer let us inspect the workflow as the user had seen it. I describe the implementation in How I Built Tooling to debug AI agents in production.
In Shraddha's manufacturing example, imagine trying to reproduce the agent's original recommendation later that morning. At 8 a.m., it pointed to the recently serviced machine. At 11 a.m., the same request gets an explanation of the changed test limit. Nobody has changed the model or the prompt.
The difference is in the records available to it. The new test results arrived overnight, but the updated test-program information did not reach the analytics system until 10 a.m. The first report combined current results with an out-of-date account of how those results had been judged. Every data request completed successfully.
Without a record of the first run, the team might assume the agent is inconsistent or conclude that a prompt adjustment fixed it. To explain what happened, we would need the test versions and release notes it actually received, the batches included in its comparison, and the dates each source was updated. We would also retain the recommendation and the evidence shown to the engineer.
That record changes what the team fixes. It can make the test-program version available alongside each result and flag an analysis when the corresponding information is missing. It can then rerun the original case to see whether the agent asks for the missing criteria instead of attributing the drop to the machine. A model upgrade may still be worth testing, but the team now knows which failure it needs to address.
A June 2026 study by Wei Wu documents how difficult some of these failures are to detect. Over eight weeks, the researcher recorded 22 incidents in one continuously running agent runtime with 4,286 unit tests and 827 governance checks. Roughly 70% of its documented silent failures were discovered through human observation of the output rather than tests or audits.
Wu calls one mechanism fail-plausible: the model turns an underlying error into a convincing explanation, leaving the user without an actionable indication of the problem. These findings describe one runtime, rather than an industry-wide failure rate. They illustrate why an investigation needs to include what reached the user, even when individual components appear to have worked.
A useful record also helps test a model upgrade. If the original input and expected outcome are available, the team can compare models on the failure it actually observed. Without that evidence, an improved answer on a later run may reflect new data or a different request, rather than a more capable model.
Deciding what to change
Sometimes the investigation will point to the model. It may have received the necessary context and clear instructions, yet repeatedly failed to reason through the task. A stronger model may then be worth the additional cost or wait. The three questions give us a way to make that decision with evidence and to check whether the upgrade helps.
In other cases, the work is in the surrounding system. A workflow copilot needs the failed input attached to its request. A manufacturing assistant needs to know which test criteria produced the results it is comparing. A user who asked for an explanation needs a chance to decide whether anything should be changed. These are specific problems a team can fix and evaluate.
What we want from our agents is practical: less time spent assembling information and doing routine work, with enough visibility to assess the result before relying on it. My users need to understand and recover from changes to their workflows. Shraddha's engineers need to examine the evidence behind a production recommendation. Both depend on more than the quality of the model's final answer.
Before changing models after the next failure, first trace the request through those conditions. What information was available? What outcome was expected? What actually happened? Once we can answer those questions, we can choose a fix, check it against the original failure, and give users a better reason to trust the next result. Otherwise, we may just be paying more for the same mistake.
Related:
First published with Shraddha Bodhe on Before You Blame the Model (AI from the Trenches), 2026-10-22.
