LLM Evaluation Engineer Interview: Metrics, Human L...

An interviewer asks, “Which metric would you use to evaluate this assistant?” The tempting answer is to name accuracy, win rate, or an LLM judge. The stronger answer is another question: “What decision must this evaluation support, for which users, and which failure would make the release unacceptable?”

Evaluation engineering is experimental design under product constraints. A defensible plan connects the decision, dataset, rubric, labels, judge calibration, slices, uncertainty, and release threshold before presenting one score.

Start With the Decision, Not the Metric

Define what will change because of the evaluation. Are you choosing between prompts, approving a model upgrade, setting an automation boundary, comparing retrieval strategies, or monitoring a deployed workflow?

The decision determines the evidence. A support-answer assistant may need factual grounding, policy compliance, completeness, tone, and escalation quality. A structured extraction task may prioritize field-level correctness and schema validity. A brainstorming tool may value usefulness and diversity while tolerating more variation.

Write a decision statement: “We will expand the assistant from draft-only to automatic sending only if critical policy failures remain below the approved limit and representative user slices do not regress.” This is more useful than “We want the highest average score.”

Then identify the cost of false approval and false rejection. A strict threshold can prevent harmful releases but also block useful changes. A permissive threshold can improve speed while exposing users to unacceptable failure.

OpenAI, Anthropic, and Google Cloud publish first-party evaluation guidance that emphasizes task-specific criteria and representative examples. Their product features differ, and vendor guidance is not independent comparative proof.

Build a Representative Dataset and Slice Plan

An evaluation set is a model of expected use, not a random pile of prompts. Define the population, sampling method, time range, user segments, languages, task types, and failure cases.

Include several layers:

common successful workflows;
difficult but valid requests;
ambiguous inputs requiring clarification;
policy-sensitive or high-cost cases;
historical production failures;
rare slices with disproportionate consequence;
out-of-scope requests that should be refused or routed.

Separate development, calibration, and holdout sets when the team will tune prompts or judge instructions. Reusing the same examples for iteration and final approval can hide overfitting.

Track dataset provenance and version. A benchmark result without the dataset, sampling rules, and method cannot support a release claim.

The evidence-anchored scoring guide explains why missing evidence should not be silently converted into low performance. Apply the same discipline to evals: label an example unscorable when the reference or context is insufficient.

Make Human Labels Auditable

Human evaluation is not automatically ground truth. Reviewers can interpret vague criteria differently, miss context, become fatigued, or apply expertise unevenly.

Create a rubric with observable anchors. “Good answer” is weak. “States the refund eligibility rule, cites the authorized policy source, asks for the missing purchase date, and does not issue a refund” can be reviewed.

For each criterion, define:

unit of evaluation;
allowed evidence;
rating levels with examples;
critical-failure override;
uncertainty or Not Scorable state;
adjudication path;
reviewer qualification.

Run a pilot label set before scaling. Measure agreement, but do not hide disagreement behind one aggregate coefficient. Inspect which criteria, slices, and examples produce conflict. Sometimes the rubric is ambiguous; sometimes the task lacks enough evidence; sometimes expertise genuinely changes the judgment.

Use adjudication to improve the rubric, not merely force one final label. Preserve the original labels and reasons so future teams can understand why the definition changed.

Combine Deterministic Checks With Quality Rubrics

Some behavior should be tested deterministically. Schema validity, required citations, tool authorization, forbidden fields, exact extraction values, and latency or cost budgets may have explicit checks.

Other behavior needs judgment. Helpfulness, completeness, clarity, grounded synthesis, and appropriate uncertainty may require human or model-based review.

Avoid one blended score that hides a critical failure. A response can earn high style and relevance scores while fabricating a policy citation. Use hard gates for unacceptable failures and separate quality dimensions for trade-offs.

The guardrails and evals interview guide provides a broad candidate framework. An evaluation engineer should go deeper: define how each metric connects to the decision, what uncertainty remains, and which failure cannot be averaged away.

Report denominators and confidence. “Ninety percent passed” means little without sample composition, repeated-run policy, judge method, and critical-failure count.

Calibrate LLM-as-a-Judge Against Humans

An LLM judge can apply a rubric at scale, compare outputs, or produce structured reasons. It is still a model with sensitivity to wording, order, context, and its own preferences.

Build a calibration set with high-quality human review. Include clear passes, clear failures, ambiguous cases, long and short answers, different styles, and slices where the judge may share bias with the system being evaluated.

Measure more than agreement. Check false approvals on critical failures, false rejections on valid variation, positional bias in pairwise comparison, sensitivity to answer length, consistency across repeated runs, and disagreement by slice.

When human and model labels disagree, do not declare the model or human correct automatically. Review the evidence, rubric, and context. The result may reveal a judge failure, label error, ambiguous task, or missing reference.

LLM-as-a-Judge is not ground truth and does not replace subject-matter experts for high-stakes cases. Use it within a calibrated system with sampled human review and a path for uncertainty.

Set Release Gates, Slices, and Regression Rules

A release decision needs more than a higher average. Define required improvement, non-regression constraints, critical failure limits, and uncertainty handling before running the comparison.

NIST AI RMF 1.0 offers a voluntary risk-management frame and is under revision at publication. It can organize governance questions, but it does not certify a judge, metric, vendor, or release decision.

For example:

primary task quality must improve on the holdout set;
no critical policy failure may exceed the approved limit;
priority language and customer slices cannot regress beyond tolerance;
tool-call validity and citation checks must pass;
latency and cost must remain inside the product budget;
inconclusive results trigger more evidence, not optimistic approval.

Monitor production drift after release. User requests change, knowledge sources age, model behavior shifts, and judge calibration can decay. Retain a governed set of failure examples and add new production cases without turning the benchmark into a collection of one-off patches. For conversational products, the Voice AI engineer guide shows why endpointing, interruption, partial state, and recovery need their own slices.

The agent observability interview guide shows how traces and failure receipts can connect a bad outcome to the stage that produced it. Observability supplies examples; evaluation determines whether the behavior meets a defined standard.

Explain the Evaluation Receipt

An evaluation result should be reproducible enough for another reviewer to understand what was tested. Record dataset version, prompt and model versions, tool configuration, rubric, judge instructions, reviewer pool, randomization, run count, slice results, exclusions, and decision.

Protect sensitive content. Store references and redacted examples where full production prompts or user data are unnecessary. Restrict access and retention by purpose.

The AI interview audit-trail guide demonstrates why prompts, diffs, tests, and decisions belong in one reviewable chain. In an eval program, add dataset and label lineage.

When reporting, lead with the decision and boundaries: “The candidate model improved task completion on the holdout set, but the multilingual support slice regressed and the result is not approved for broad release.” This is more credible than selecting the highest favorable score.

Answer the Interview With One Complete Experiment

Use a seven-part response: decision, population, dataset, rubric, label plan, judge calibration, and release rule.

A concise answer might say: “I would first define whether the eval chooses a prompt or authorizes a broader automation boundary. I would build a versioned dataset with common tasks, critical failures, historical incidents, and priority slices. Deterministic checks would cover schema, citations, and tool policy; humans would label task quality with an anchored rubric and adjudication. I would calibrate an LLM judge on reviewed examples, measure critical false approvals and slice disagreement, and keep sampled human review. The release gate would require primary improvement, no unacceptable regression, and sufficient evidence.”

Then state limits. Vendor evaluation tools can implement parts of the workflow, but they do not prove one design is better. A high judge score is not ground truth. A benchmark gain without method and dataset should not enter the conclusion.