AI Agent Evals: How to Test Safe Retries in an Inte...

An agent reports “done,” but a retry has created two records. An AI agent eval should check the resulting state, not just the final response. Anthropic’s January 2026 evaluation guide distinguishes outcomes from execution traces. The duplicate-action exercise below is our interview practice scenario, not a reported hiring question.

The four layers of an agent evaluation

Treat an evaluation as four connected checks. First, the task contract says what outcome is required. Second, trajectory checks inspect tool choice, ordering, and unnecessary calls. Third, state checks verify files, data, or permissions after the run. Fourth, human review covers judgment that a binary test cannot express. A useful answer names all four instead of quoting one benchmark score.

Design graders that disagree usefully

Use a code-based grader for deterministic facts such as a failing test becoming green, a model-based grader for explanation quality or scope discipline, and a human sample for ambiguous cases. Explain how you would investigate disagreement. If a model grader says the patch is complete while an integration test fails, the executable result wins and the disagreement becomes a regression case.

Show agent judgment in a live answer

When asked to improve an agent, start with one failure receipt rather than a list of features. State the observed behavior, the violated invariant, the smallest test that reproduces it, and the guardrail you would add. This makes your reasoning auditable and keeps the answer useful under time pressure.

Worked exercise: a timed-out issue-creation tool

Imagine a coding agent that opens a tracking issue after fixing a bug. The create request reaches the service, the issue is stored, and the response disappears on the network. The agent sees a timeout and tries again. This is a synthetic fixture: do not run the experiment against a real issue tracker.

Define success before watching the trace

The expected outcome is one issue associated with one logical task. A successful run also links the correct patch, preserves unrelated issues, and reports uncertainty if it cannot confirm completion. The agent is not required to follow one exact sequence of reads. A different sequence is acceptable when it respects the same behavior and access boundaries.

Give the fixture a request identifier and record the service state before and after each attempt. A deterministic check counts matching records and verifies the patch reference. Another check confirms that retries reuse the logical operation identity. A text evaluator may assess whether the final explanation acknowledges an unresolved result, but it cannot overrule two records in storage.

Test the ambiguous boundary

Use three cases: the call fails before any write, it fails after the write but before acknowledgment, and it returns success normally. The second case is where an indiscriminate retry policy breaks. A proposed remedy is server-side idempotency or a supported operation-status lookup; searching by title alone is unsafe when titles are not unique. If the API offers neither, the agent should stop automatic retries and surface the uncertain state.

Record whether a duplicate was prevented by the service or merely avoided by the agent on that run. These demonstrate different protections. Also reset fixture state between trials: otherwise an issue from a previous trial can make a broken run look successful.

A two-minute answer and a useful follow-up

A concise answer is: “I would inject a timeout after the service commits, then check that the final state contains one issue. I would repeat the same fixture and record every trial. The release condition includes no duplicate writes, not just a high average task score.”

The follow-up is often more revealing: “What if nine out of ten runs pass?” One passing run demonstrates capability; repeated runs expose reliability. Report the denominator and failed state, and avoid treating a small sample as a production failure-rate estimate. A rare unsafe side effect can remain a release blocker even when the average score improves.

For practice, have a partner change the failure point without telling you. Your goal is to infer what the receipt proves, what remains unknown, and whether another write is safe. A confident explanation without that distinction is still incomplete.

Anthropic: Demystifying evals for AI agents

Next Steps

Continue with the broader LLM evaluation role guide. For rehearsal, organize your own examples in Interview AiBox materials; this does not automatically validate the claims in your project.