Design an LLM Evaluation System: The Interview Round Where 'It Works' Is Not an Answer

The question usually comes near the end of the loop, and it is phrased casually enough that candidates relax when they hear it. "So — how would you actually know if this is working?" A candidate who just spent forty minutes designing a retrieval pipeline or an agent hears a wind-down question and answers accordingly, with a sentence about accuracy and a mention of user feedback. The interviewer writes a note, and the note is rarely kind.

It is not a wind-down question. In an LLM system design loop it is frequently the highest-signal question asked, because it is the one a candidate cannot answer from having read about the architecture. Evaluation is the part of the work you only understand once you have shipped something, watched it look fine in a demo, and then watched it fail in production in a way no offline number predicted. The round is built to find out whether that has happened to you yet.

The reason evaluation carries so much signal in 2026 is that model quality has largely commoditized. Anyone can call a frontier API and get a fluent, well-formatted, confident-sounding answer. Fluent and confident is precisely what these models are trained to produce, whether or not the content is correct. The hard engineering problem is no longer generating a plausible output. It is knowing, at scale and continuously, what fraction of those plausible outputs are actually right, which kinds are wrong, and whether the change you shipped this morning made any of it better or worse. That is evaluation, and it is the discipline most engineers were never taught, because for most of machine learning's history evaluation meant a held-out test set and a single accuracy number.

Scope the eval before you design it

The first move is to refuse the generic answer, because the right eval strategy depends entirely on what is being evaluated. There is a spectrum, and where a product sits on it determines everything downstream.

At the easy end sits classification and extraction: sentiment labels, routing decisions, structured field extraction. Ground truth exists, the output space is small, and the classical metrics apply almost unchanged. At the hard end sits open-ended generation and multi-step agents: summaries, chat, code changes, research tasks. There is no single correct output, the space of acceptable answers is enormous, and the difference between a good answer and a subtly wrong one is often a matter of judgment rather than a string match.

A strong candidate states this distinction unprompted and locates the product on it, because it decides whether evaluation is a solved problem with known metrics or an open one requiring judges, human labels, and a great deal of care. The second move is to enumerate the failure modes that actually matter for this product. For a customer support assistant the expensive failure is a confident wrong answer about a refund policy. For a code agent it is a change that passes tests and breaks production. For a summarizer it is a fact that appears in the summary but not the source. Evaluation is the measurement of specific failure modes, and a candidate who cannot name the failures for their product cannot design a system to catch them.

Two evals, measuring two different things

Every serious answer separates offline evaluation from online evaluation early and keeps them separate, because they answer different questions and one is routinely mistaken for the other.

Offline evaluation runs a fixed set of inputs through the system and scores the outputs, before anything ships. It is fast, cheap, repeatable, and completely under your control, which is exactly why it is also a controlled fiction: it measures performance on the inputs you thought to include, under conditions you constructed. Online evaluation measures what the system actually does to real users on real traffic, which is the thing you care about, at the cost of being slow, noisy, expensive, and impossible to run before you have already exposed users to the change.

The gap between the two is where LLM products live and die. A change that improves every offline number can degrade the live experience, because the offline set did not contain the inputs the change made worse. A change that looks flat offline can win online because the offline set underweights the cases it helps. Candidates who treat an offline score as proof of anything, with no account of how it connects to online outcomes, are describing a workflow they have not run end to end. The honest framing is that offline eval is a cheap filter that decides what is worth testing online, and online eval is the source of truth that decides what ships.

The golden set, and the fact that it rots

Offline evaluation needs a dataset, and the construction and maintenance of that dataset is more of the work than candidates expect.

A golden set is a curated collection of inputs paired with known-good outputs or graded rubrics. It has to be stratified deliberately: common cases in proportion, but also a heavy overweight on the edge cases and known failure modes, because the average input is not where the system breaks. It has to be large enough to produce a stable signal and small enough to run cheaply on every change, which is a tension resolved by having tiers — a small fast set for every commit, a large comprehensive set for release gates. Part of it can be generated synthetically, using a model to draft candidate inputs from real data, but the outputs a synthetic set is graded against need human verification or the whole exercise measures agreement with a model rather than correctness.

The point most candidates miss is that the golden set decays. Production traffic drifts, new features change the input distribution, and last quarter's edge cases stop being the ones that matter. A golden set frozen at launch is measuring a product that no longer exists. The strong version of the answer treats the set as a living artifact, continuously fed by mining real production failures and adding each new class of bug as a permanent regression case. The best signal a candidate can give here is describing the loop where a production incident becomes a golden-set entry that can never regress again, which is the eval equivalent of a regression test and the mechanism by which the same bug never ships twice.

LLM-as-judge, and why you cannot trust it naively

For open-ended outputs there is no string to match against, so the dominant technique is to use a strong model as the grader. It scales, it is cheap relative to human labeling, and used carefully it correlates well with human judgment. Used carelessly it manufactures numbers that feel like measurement and are not.

The failure modes are well documented and an interviewer expects a candidate who proposes an LLM judge to name them. Judges exhibit position bias, systematically favoring whichever answer is presented first in a pairwise comparison. They exhibit verbosity bias, preferring longer, more elaborate answers regardless of correctness. They exhibit self-preference, scoring outputs from their own model family more highly. They can be swayed by formatting and confident tone, the exact surface features that are decoupled from correctness. And there is a circularity problem when the judge and the system under test share training lineage, because the judge's blind spots and the system's blind spots overlap.

The mitigations follow directly. Control position bias by running each comparison in both orders and averaging, or by discarding pairs where the verdict flips when the order flips. Prefer pairwise comparison over absolute scoring, because models are far more reliable at "which of these two is better" than at "rate this from one to ten." Pin the judge with a concrete rubric rather than a vague instruction to assess quality. And most importantly, calibrate the judge against human labels on a sample and measure the agreement rate explicitly, so the judge's reliability is a known quantity rather than an assumption. A judge you have not validated against humans is not an evaluator. It is a second model whose errors you have chosen not to look at.

Metrics are plural, and the interesting ones are decompositions

The candidate who reaches for a single quality score has usually not operated one of these systems. Quality decomposes, and the decomposition is where the useful signal lives.

The clearest example is the split between relevance and faithfulness in any retrieval-grounded product. Relevance asks whether the answer addresses the question. Faithfulness asks whether the answer is actually supported by the retrieved evidence rather than invented. These fail independently: an answer can be perfectly on-topic and quietly fabricated, and measuring them as one number hides exactly the failure that matters most. The same decomposition logic applies across products — helpfulness against harmlessness, completeness against conciseness, correctness against calibration. Capability evaluation, which asks whether the system does the job, is a different axis from safety evaluation, which asks whether it refuses the things it should and resists adversarial inputs, and a mature eval system runs both because a launch can improve one while regressing the other.

The framing that lands is that you evaluate the dimensions that fail independently, separately, so that a regression in one is visible rather than averaged away by a gain in another.

Eval in the loop: treat prompts like code

The shift a senior candidate describes without being asked is that in a mature LLM product, evaluation is not a phase that happens before launch. It is continuous integration.

Every prompt edit, model version bump, retrieval change, and parameter tweak is a deploy that can silently regress behavior, and a one-line change to a prompt can move quality as much as a model swap. The discipline that answers this is running the offline eval suite automatically on every change, with a quality gate that blocks a merge when key metrics regress past a threshold, exactly as a test suite blocks a merge on a failing test. Taken to its conclusion this becomes eval-driven development: you write the eval cases that define acceptable behavior before you write the prompt, and you iterate the prompt against them, which inverts the usual order in which people ship a prompt that looks good in a handful of manual trials and discover the failure distribution in production.

A candidate who describes eval running in CI, gating deploys, has demonstrated that they treat an LLM system as a production artifact under change control rather than a prompt someone tuned by hand until the demo looked good.

Online evaluation without a ground truth

Production is where the truth is and also where the ground truth is missing, because in the live system nobody has labeled the correct answer. This is the part of the answer that most separates people who have run these systems from people who have not, and it comes down to constructing signal where none is handed to you.

Implicit signals come first: did the user accept the suggestion, copy the answer, retry the query, rephrase in frustration, escalate to a human, abandon the session. None of these is a clean label and each is a proxy, but in aggregate they track quality closely enough to alarm on. Explicit signals, thumbs and ratings, are sparse and biased toward the annoyed, useful in volume and dangerous as an unweighted average. On top of these sits continuous sampling: a fraction of production traffic is pulled, run through the LLM judge, and a smaller fraction escalated to human review, which turns the live system into a permanent source of graded examples and, not coincidentally, of new golden-set cases. Drift detection watches the input distribution and the output distribution for movement, because the earliest sign of trouble is often that traffic has shifted into a region the system was never evaluated on. And every launch is gated on an A/B test read on guardrail metrics, not only the primary one, so that a change which improves the target while quietly degrading latency, cost, or safety is caught before full rollout.

The human layer is the anchor, so spend it well

Underneath the entire structure, humans remain the ground truth. The LLM judge is calibrated against human labels. The golden set outputs are verified by humans. The ambiguous production samples are adjudicated by humans. Every automated layer in the system ultimately traces its authority back to human judgment, and a design that never mentions people has quietly assumed away the thing that makes the numbers mean anything.

Because human labeling is the expensive resource, the engineering problem is spending it where it buys the most: on calibrating the judges that then scale to everything else, on the ambiguous cases where automated scores disagree or sit near a threshold, and on the new failure modes that no existing rubric covers. Active-learning selection of the highest-value examples to label, rather than uniform random sampling, is the mark of a candidate who has actually paid for annotation and felt the budget. The relationship to state clearly is a hierarchy: humans calibrate judges, judges scale to sampled traffic, and the golden set captures everything either one has already caught, so that human attention is concentrated on the frontier of what the system does not yet know how to grade.

Where answers fall apart

A handful of failure modes account for most weak responses to this question. Offering accuracy, or any single quality number, as though correctness for open-ended generation were a solved measurement. Presenting an offline improvement as proof of success with no online evaluation behind it. Proposing an LLM judge with no acknowledgment of its biases and no calibration against humans. Describing a golden set as a fixed asset rather than a decaying one that must be continuously fed from production. Treating evaluation as a launch checklist rather than a gate that runs on every change. And omitting the human layer entirely, which leaves a tower of automated metrics with nothing anchoring it to reality.

The question is deceptively small and the answer is a system in its own right, with an offline path and an online path, a judge layer calibrated against a human layer, a golden set that learns from every incident, and a gate that stands between every change and production. Building the product that the eval measures is, in 2026, the easy half. Knowing whether it works — continuously, honestly, at scale, as it changes — is the half that is actually hard, and it is the half the round exists to test. Candidates who have lived on the wrong side of a confident wrong answer in production tend to answer it well. Candidates who have only ever watched the demo tend to reach for accuracy, and the note the interviewer writes is rarely kind.