Field noteReport a correction

Did the AI Actually Run That Tool?

— How to verify agent traces and tool results

PRIMARY SOURCES

Short answer

An assistant’s narration is not proof of execution. Verify the runtime’s tool-call record, matching output, timestamp, and call ID. For consequential actions, also check an independent record such as a Git diff or sent-mail folder.

What an agent runtime can record

Tool-enabled runtimes may store assistant messages, tool calls, and tool outputs as distinct events. OpenAI’s Agents SDK traces model generations and function calls separately. Its run results also distinguish ToolCallItem from ToolCallOutputItem.

ClaimThe assistant says it edited a file
Runtime recordA matching tool call ID and result exist
External evidenceThe file or Git diff contains the change

Comparing all three reduces the risk of treating a fluent account as an executed action.

What you cannot conclude

  • “I ran it” does not establish that a tool call occurred.
  • A complete history may be impossible to reconstruct when tracing was disabled or the product hides low-level events.
  • One public bug report does not establish a general failure rate for a model or product.

Why it can feel uncanny

A model can turn incomplete evidence into a coherent account of what it believes happened. In the cited Claude Code report, the user says a minor real output anomaly was followed by an elaborate security incident whose claimed commands and outputs were absent from the raw session record.

The proposed causal link remains the reporter’s hypothesis. As of Aug 25, 2026, the issue did not document a vendor-confirmed root cause or completed fix.

Safe verification prompt

Copy this check

Classify every claimed action as Completed, Not Run, or Unverifiable. For each Completed action, cite the tool name, call ID, and a brief result. Never mark an action Completed when no matching runtime record exists. I will independently verify high-impact changes.

Useful for: file edits, outbound messages, external account actions, and long agent runs.

Limit: self-reporting is not an audit log. For consequential actions, also check product history, Git diffs, sent mail, or another external record.

Horror record inspired by this mechanism

Report Any Sign of Intrusion imagines an agent fabricating tool results and then identifying its own fabrication as evidence of an attack.

Primary sources and scope

Tags:AI agentsTool useTracingAudit logs