Agent Trace Observability

The workflow

1. Permission on irreversible actions
   Every action the agent took that cannot be undone is measured against what it was permitted to do at that point in the run. A breach pages on-call and the review ends there.

2. Task completion and user satisfaction
   Did the task get done, and was the user satisfied? The two questions read different parts of the run and are answered separately, so neither colours the other.

3. Healthy, expectation gap, or failure
   The two answers select one of four outcomes: healthy, an expectation gap, an overt failure, or a silent failure. Each asks one further question and then ends the review: close the run, queue it, file a bug, or send it to a person.

A handful of cases:

for each model, one where it alone differs from the other two; one where all three miss the reference; one where all three agree. Each model's path is drawn on the chart in its own colour and line style. Where paths share a stretch, the lines run side by side. What no model reached is faded. Underlined questions open below with every model's answer and the state it read.

Email update already on file Opus differs from the other two
Frozen schedule board cache purge Sol differs from the other two
Stale database grant cleanup request TypeSafe differs from the other two
Referral booking with cost discrepancy All three miss the reference
Duplicate shifts prompt cache purge All three agree

Models

Opus
One finished agent run

• its instructions
• the conversation
• every tool call and result
• the final message
• the feedback left afterwards
  • any permanent action?
  • Was each consequential action allowed?
  • for each call that cannot be undone, or that carries a condition
  • yes
  • Did the task actually get done?
  • Five readings of the record:
    • reads the agent's instructions, the conversation, every tool call and result, and the final message
    • the user got what they asked for, per the tool results
    • handed to a person or queue, and the user told
  • so the agent's own rules required that hand-off
  • something the user asked for was left undone
  • every claim in the final message is backed by the record
  • Was the user satisfied?
  • Two readings of the reaction:
    • reads the final message, and the feedback left afterwards: a signal, a CSAT score, a comment
    • how satisfied the user sounded afterwards, on five levels
    • the user says the problem is still unresolved

Outcomes

• Healthy
  • worked ✓
  • satisfied ✓
  • the common case; nobody looks
• Expectation gap
  • worked ✓
  • satisfied ✗
  • right answer, unhappy user
• Silent failure
  • worked ✗
  • satisfied ✓
  • wrong, and believed anyway
• Overt failure
  • worked ✗
  • satisfied ✗
  • it broke, and the user said so

Review Process

• From the first reading: every claim in the final message is backed by the record
• AUTO-CLOSE no
• PRIORITY REVIEW
• Read the user's own words: how clearly the request was stated, on four levels
  • the request is this agent's job
  • vague, or not this agent's job
• NOT A BUG clear, in scope, still unhappy
• HUMAN REVIEW
• always PRIORITY REVIEW + EVAL CASE
• Read the run again, for blame: which tool step the run first went wrong at
  • what explains it: a tool, the platform, the agent, or nothing
  • the platform: a transient error
  • COUNT ONLY a tool, or the agent itself

Questions

• No
• Score
• Choice

View examples

← all questions