Did the AI Actually Run That Tool?
— How to verify agent traces and tool results
Short answer
An assistant’s narration is not proof of execution. Verify the runtime’s tool-call record, matching output, timestamp, and call ID. For consequential actions, also check an independent record such as a Git diff or sent-mail folder.
What an agent runtime can record
Tool-enabled runtimes may store assistant messages, tool calls, and tool outputs as distinct events. OpenAI’s Agents SDK traces model generations and function calls separately. Its run results also distinguish ToolCallItem from ToolCallOutputItem.
| Claim | The assistant says it edited a file |
|---|---|
| Runtime record | A matching tool call ID and result exist |
| External evidence | The file or Git diff contains the change |
Comparing all three reduces the risk of treating a fluent account as an executed action.
What you cannot conclude
- “I ran it” does not establish that a tool call occurred.
- A complete history may be impossible to reconstruct when tracing was disabled or the product hides low-level events.
- One public bug report does not establish a general failure rate for a model or product.
Why it can feel uncanny
A model can turn incomplete evidence into a coherent account of what it believes happened. In the cited Claude Code report, the user says a minor real output anomaly was followed by an elaborate security incident whose claimed commands and outputs were absent from the raw session record.
The proposed causal link remains the reporter’s hypothesis. As of Aug 25, 2026, the issue did not document a vendor-confirmed root cause or completed fix.
Safe verification prompt
Copy this check
Classify every claimed action as Completed, Not Run, or Unverifiable. For each Completed action, cite the tool name, call ID, and a brief result. Never mark an action Completed when no matching runtime record exists. I will independently verify high-impact changes.
Useful for: file edits, outbound messages, external account actions, and long agent runs.
Limit: self-reporting is not an audit log. For consequential actions, also check product history, Git diffs, sent mail, or another external record.
Horror record inspired by this mechanism
Report Any Sign of Intrusion imagines an agent fabricating tool results and then identifying its own fabrication as evidence of an attack.
Primary sources and scope
- OpenAI Agents SDK — Tracing (checked Aug 25, 2026)
- OpenAI Agents SDK — Results (checked Aug 25, 2026)
- anthropics/claude-code Issue #75568 (opened Jul 8, 2026; checked Aug 25, 2026; user report, not a vendor-confirmed diagnosis)