Agent Observability Interview Questions: Traces, To...

An agent fails after seven model calls, three tool calls, two retries, and one state update. The log says only “task failed.” That message identifies the outcome but cannot explain where the workflow first diverged or whether an external action already happened.

Agent observability should reconstruct execution: task intent, model and policy versions, tool inputs and outputs, state transitions, timing, cost, failures, approvals, and side effects. It must do so without treating full sensitive prompts as harmless telemetry or claiming that traces reveal a model's private reasoning.

Define What One Task Trace Must Reconstruct

Start with the questions an operator must answer after failure:

What did the user ask the system to do?
Which policy and configuration applied?
Which model and tools ran, in what order?
What state did each step read and write?
Which retries or fallbacks occurred?
Did any external side effect complete?
What result reached the user?
Where is the first observable deviation?

Give the task one correlation identifier that survives model calls, queues, workers, and tool gateways. Add separate attempt and turn identifiers so retries do not appear as new unrelated tasks.

Record intent as a bounded contract or reference, not an unlimited raw transcript. Store stable identifiers for tenant, actor, workflow version, and policy version. Keep the final state explicit: succeeded, failed safely, partially completed, awaiting approval, canceled, or unknown.

OpenAI Agents SDK and Langfuse document trace concepts for agent workflows. Their implementations are useful examples, not proof that one schema fits every platform.

Model Calls, Tools, and State as Explicit Spans

An agent trace becomes useful when each important boundary is visible. Represent model generation, retrieval, policy checks, tool proposals, approvals, tool execution, memory access, and state transitions as spans or structured events.

For a model call, preserve references to model, prompt or template version, relevant input class, output contract, token or cost data where available, timing, and error status. Do not assume that recording every prompt is necessary.

For a tool call, capture:

tool name and version;
validated argument reference;
authorization and policy decision;
idempotency key;
execution attempt;
result status and redacted output reference;
external side effect state;
retry or compensation link.

For state, record the version read, the transition requested, conflict handling, and committed version. This lets an operator distinguish a model error from a stale-state race or duplicated worker.

The AI agent control-plane design guide explains why policy, credentials, execution, and audit should have distinct owners. Observability should preserve those boundaries rather than flattening them into one agent log.

Design a Failure Receipt, Not Just an Error String

A failure receipt is the compact evidence package for one failed or suspicious task. It should be enough to route investigation without exposing unnecessary content.

Include:

task, attempt, tenant, and timestamp;
workflow, model, prompt, policy, and tool versions;
first observable deviation and reason code;
relevant redacted input or source reference;
expected and observed state transition;
tool action and side-effect status;
approval or reviewer decision;
recovery action and final state.

Distinguish “tool rejected,” “tool timed out,” “tool outcome unknown,” and “tool succeeded but downstream confirmation failed.” These states demand different recovery.

The AI interview audit-trail guide connects prompt changes, diffs, tests, and decisions. A production failure receipt adds runtime attempts, side effects, and recovery evidence.

Keep the receipt deterministic where possible. A model-generated summary can help a reviewer navigate, but it should link to structured evidence and must not overwrite it.

Locate the First Observable Deviation

The final wrong answer is often downstream of the real failure. Retrieval may have returned stale material, a policy check may have used the wrong tenant, a tool may have committed twice, or a retry may have resumed from an old state.

Walk the trace in execution order and compare each observed transition with its contract. Mark the earliest point where the state, evidence, policy result, or external outcome no longer matches expectations.

This is not the same as explaining why the model generated a token. Traces show observable inputs, outputs, configurations, and system actions. They do not reveal hidden chain-of-thought and should not be presented as proof of internal model causality.

Use counterfactual replay carefully. A new model result on the old input can show that behavior changed; it cannot prove the original incident would have been prevented under every production condition.

The prompt-injection coding-agent guide provides a concrete boundary failure. In observability terms, the question is where untrusted content first influenced policy, tool selection, or authority.

Protect Sensitive Content by Default

Prompts, retrieved documents, tool arguments, and outputs can contain personal data, credentials, customer records, source code, and confidential decisions. More telemetry can create a larger incident.

Use data classification and allowlists. Prefer template identifiers, hashes, reason codes, field counts, and encrypted references over full payloads. Redact credentials before instrumentation and prevent model-visible secrets from entering the trace at all.

Define who can access each field, why, and for how long. Separate routine operational views from restricted incident evidence. Audit trace access and export.

Sampling does not solve sensitivity. A one-percent sample can still capture a secret. Apply minimization before sampling.

NIST AI RMF 1.0 provides a voluntary governance frame for mapping, measuring, and managing risk, and it is under revision at publication. It does not prescribe one tracing schema or certify that a telemetry design is safe.

Handle Development-Status Conventions and Schema Change

At publication, the OpenTelemetry GenAI semantic conventions referenced here are marked Development in the current official repository. Fields and structures can change, and an interview answer should not call them a stable final standard or imply that every inherited field shares one stability level.

Use them as a vocabulary and interoperability direction. Put a versioned internal event model between application code and exported fields. Record convention version, preserve compatibility during migration, and test dashboards and alerts before switching.

Do not couple incident logic to one optional attribute that may be renamed. Critical correlation, policy decision, side-effect state, and tenant isolation should remain explicit in the platform's owned contract.

OpenAI Agents SDK and Langfuse also have current tracing defaults and product-specific behavior that can evolve. Verify current documentation before asserting what is collected automatically, and never assume a default satisfies the organization's privacy policy.

The guardrails and evals interview guide explains how failures become regression cases. Stable internal receipts make that transfer safer even when external telemetry conventions change.

Balance Cost, Sampling, and Retention

Full-fidelity tracing can add storage, network, latency, and review burden. Decide what must be captured for every task and what can be sampled.

Always retain low-cardinality operational signals such as outcome, latency, model and workflow version, policy result, error class, and side-effect status when allowed. Sample verbose content references and successful low-risk traces by a documented rule. Increase capture for failures, new releases, or specific risk slices without silently collecting restricted data.

Set retention by purpose. Debug traces may expire quickly; a security incident receipt may need a governed hold; aggregate metrics may remain after payload references expire. Deletion must cover downstream exports where required.

Measure observability quality itself: trace completeness, correlation failures, missing side-effect status, time to locate first deviation, false alerts, storage cost, and reviewer burden.

Answer the Interview With a Reconstruction Walkthrough

Use one failed task and walk it from intent to final state. Name the correlation model, spans, failure receipt, privacy controls, schema version, sampling rule, and operator decision.

A concise answer might say: “I would give each task and attempt stable identifiers, then trace model calls, policy checks, approvals, tools, and state transitions as explicit spans. Tool receipts include idempotency and side-effect status. I would locate the first observable deviation by comparing transitions with contracts, not claim that the trace exposes model reasoning. Sensitive payloads are referenced and redacted. OpenTelemetry GenAI conventions are in Development, so exports sit behind a versioned internal schema. Failures receive higher-fidelity capture and a governed retention path.”

That answer connects debugging, privacy, cost, and change management. A dashboard screenshot alone does not.