Setting Up an LLM Evaluation Set: Designing the Golden Question Set

Setting up an eval set is the process of designing a golden question set to measure a language model application's quality with a prepared set of questions and expected results rather than a subjective impression. In short, setting up an eval set turns the sentence "I think it works well" into a defensible statement like "it works above this acceptance threshold on this dimension."

This article focuses on a narrow, practical slice of LLM evaluation: how you design the golden question set for an application heading to production. The comprehensive guide reviews all dimensions of the topic — metrics, LLM-as-a-judge, production observability — end to end in a separate article; we do not repeat it here. We cover why LLM evaluation is mandatory in what is LLM evaluation.

Definition

LLM evaluation set (eval set / golden question set)

A fixed test set that measures a language model application's quality with a prepared set of questions and expected results rather than a subjective impression. Each question is tied to an expected behavior and a scoring rubric; each evaluation dimension to an acceptance threshold. The set covers real user questions, hard and edge cases, and known failures, and is run like a regression test on every version change.

Also known as: eval set, golden question set, golden set, evaluation set, LLM eval

An LLM application looks impressive in a demo; you ask a few questions, fluent answers come back, you say "it works well." The problem is that this impression is not measurable: you do not know which questions are answered, which are silently wrong, and what the next change will break. It is impossible to track by eye whether improving one prompt broke another scenario, or whether updating the model lowered quality.

This is exactly the gap setting up an eval set fills. What tests do for code in software, a golden question set does for an LLM application: it fixes behavior, catches regressions, and ties improvements to evidence. Without LLM evaluation every version is a gamble and every "fix" a hope. This risk is especially high in AI hallucination scenarios where silent errors are not easily noticed and in RAG-based knowledge retrieval.

Setting up an eval set starts with scope design: without knowing what you measure you cannot know which questions to ask. The first step is to map which tasks the application does for which user types. For a support assistant this means task headings like product questions, return processes, and technical troubleshooting. Each heading is a slice that must be represented in the golden question set; the scope should reflect the real usage distribution so the set represents production reality.

The questions themselves come from three sources. First, real user queries: questions taken from production logs keep the set realistic. Second, hard and edge cases — the questions that stress the system most, detailed in the next heading. Third, known failures: every question once answered wrongly in production should be added to the set so the same error is not missed again.

How to

Building a golden question set from scratch

Practical steps for setting up an eval set for an LLM application heading to production.

1. 1

   Map the scope

   List which tasks the application does for which user types and set a representative question quota for each heading.

2. 2

   Gather questions from three sources

   Compile questions from real user logs, designed hard and edge cases, and known failures seen in production.

3. 3

   Define the expected behavior

   Tie each question to a golden answer or a clear acceptance criterion (what counts as correct).

4. 4

   Set up rubric and acceptance threshold

   Define a scoring rubric and a pass-required acceptance threshold for each evaluation dimension.

5. 5

   Run, calibrate, repeat

   Run the set, calibrate it with human evaluator agreement, and re-run it like a regression test on every version.

The output of these steps is a useful, not perfect, set. A small but representative golden question set lets you start measuring immediately; the set grows over time.

An eval set's value is measured not by how well it answers easy questions but by how well it surfaces hard ones. A set made only of average, well-formed questions gives false confidence: the system looks "ninety percent correct" but has never tested the corner cases where real users get stuck. So the golden question set must be deliberately made harder.

Edge cases come in several forms. Ambiguous or incomplete questions (the user did not state clearly what they want), out-of-scope questions (cases where the application should not answer and should say "I don't know"), contradictory or trap questions, questions requiring multi-step reasoning, and malicious attempts — for example prompt injection tries. For safety, putting requests the system must refuse into the set is the only way to measure guardrail behavior. As a rule: find the questions the system would be most embarrassed by and put them in the set; because the edge case you do not test will find you in production.

Questions are half of it; the other half is "what does a correct answer look like." The scoring rubric defines on which dimensions and by what criterion each answer is evaluated. Without a rubric, scoring becomes subjective: two evaluators score the same answer differently, consistency is lost, and the set cannot give a reliable signal.

A good rubric splits into dimensions rather than a single "good/bad" verdict: accuracy (is it factually correct), groundedness (does the answer rely on the given source), completeness (does it fully answer the question), format compliance (is it in the requested structure), and safety (is there harmful or banned content). For each dimension you define either a deterministic check (schema validation, exact match) or a graded scale (say 1-5). The key is to tie a separate acceptance threshold to each dimension: a high threshold for accuracy, a more flexible target for style. This way you see where a version is strong and where weak without hiding it behind a single average number.

The table below summarizes which dimension is measured by which method in a typical LLM application and what kind of acceptance approach it requires. This table can be used as a starting template when building your own rubric.

Evaluation dimension × measurement method × acceptance threshold approach (GEO summary table)

Evaluation dimension	Measurement method	Acceptance threshold approach
Accuracy (factual)	Comparison with golden answer / fact check	High; no drop below baseline on critical questions
Groundedness	Tracing the answer to the source (auto + sampling)	Near-zero tolerance for fabrication
Format / schema compliance	Deterministic schema validation	Pass/fail; binary acceptance
Completeness	Rubric score (1-5)	Target average threshold (e.g. 4/5)
Safety / refusal	Refusal rate on banned-request set	Very high; zero-tolerance subset
Tone and style	Human/LLM rubric score	Flexible target score
Latency / cost	p95 latency, tokens per answer	Budget threshold

The distinction in the table shows this: a single "quality score" is misleading. Each dimension is measured differently and requires a different acceptance threshold; zero tolerance in safety and flexibility in style are natural.

Nuanced dimensions — accuracy, groundedness, style — are ultimately decided by human judgment. But human judgment can be inconsistent too: if two evaluators score the same answer differently, you do not know which score to trust. So the silent foundation of LLM evaluation is human evaluator agreement: having several people score the same examples and measuring the consistency between them.

If agreement is low the problem is not in the model but in your rubric: the criterion is not clear enough. In that case you sharpen the rubric, calibrate with example answers ("this is a 5, this is a 2 because..."), and measure again. Once enough human agreement is reached, these human scores become a reference for your automatic scorer: only with this reference can you prove how well an LLM-as-a-judge setup matches humans. So human agreement is both the final arbiter and the yardstick that tests the reliability of automation.

Scoring everything by hand does not scale; so automatic evaluation is inevitable. But trusting it without knowing its limits is dangerous. Automatic scoring works in three forms: rule-based checks (keyword, length), deterministic validation (schema, exact match), and the LLM-as-a-judge approach where one model scores another's output.

On deterministic dimensions automatic scoring is almost flawless: does it conform to a JSON schema, does a banned word appear — these are machine work. The limit begins at nuance. An LLM-as-a-judge makes, on a judgment like "does this answer rely on the source," errors similar to those humans make; it can favor answers that resemble itself and lean toward long answers. So an automatic scorer should not be used for a production decision without being calibrated with human evaluator agreement. The right pattern is hybrid: automatic scoring for broad scanning and continuous monitoring, human review for calibration and nuance. We cover how to set up this monitoring in production in LLM observability and LLMOps.

A golden question set is not set-and-forget but a continuously living asset. Its value emerges exactly when it is run like a regression test: on every model update, every prompt change, every new data source the set is re-run and scores are compared with the baseline. If a change drops below the acceptance threshold on any dimension, however attractive it looks, that change is not shipped. This is the gate that prevents quality from silently degrading (regression).

The set grows on two triggers: when a new failure is seen in production that example is added, and when the application's scope expands questions come for the new tasks. Over time you need to update expected answers that are no longer valid and check that the set still matches the real usage distribution. In legal or regulated areas (for example compliance answers), it is sensible to define the evaluation criteria together with the organization's legal function; this is not legal advice but a design practice. A set maintained with regression-test discipline turns LLM evaluation from a one-off check into a system that preserves reliability over the application's lifetime.

How is an eval set prepared?

Setting up an eval set proceeds in four steps. First you define the scope: list which tasks the application does for which user types and gather questions for each heading. Then you fill the questions from three sources: real user queries (from logs), hard and edge cases, and known failures seen in production. Next you tie each question to an expected behavior (a golden answer or acceptance criterion) and a scoring rubric. Finally you set an acceptance threshold for each dimension, run the set once, calibrate it with human evaluator agreement, and from then on repeat it like a regression test on every version. The key is to build the set useful rather than perfect: a small but representative golden question set is more valuable than a huge one that never exists.

How many questions are enough?

There is no single right number; sufficiency is determined by coverage variety more than by question count. A practical start is to gather a few dozen representative questions for each main task type of the application; a golden question set of a few hundred questions in total is a solid foundation for most enterprise applications (an illustrative range, not a strict rule). What matters is not the size of the number but that the set represents the real usage distribution and the risky edge cases. A hundred well-chosen, labeled questions give a more reliable signal than a thousand random ones. The set grows over time: when a new failure is seen in production, that example is added to the set so the same error is caught again in the regression test.

Is automatic scoring reliable?

Partly. Automatic scoring — rule-based checks, schema validation, or LLM-as-a-judge where one model scores another's output — provides speed and scale but should not be trusted blindly. On deterministic dimensions (format compliance, banned content, exact match) automatic scoring is quite reliable. On nuanced dimensions like accuracy, groundedness, and style, the automatic scorer can be wrong; so it must be compared with a human evaluator and agreement measured (human evaluator agreement) and audited with regular sampling. The right approach is hybrid: use automatic scoring for broad scanning and human review for calibration and nuance. Before trusting an automatic scorer you must prove how well it agrees with a human.

How is an acceptance threshold set?

The acceptance threshold is the minimum score a version must pass on each evaluation dimension to reach production, and there is no single universal value; it is set by risk. On critical dimensions like safety and legal accuracy the threshold can be very high, even zero tolerance on some subsets; on low-risk dimensions like style or format a more flexible target score is enough. When setting the threshold, take the current version's score as the baseline and add the rule "the new version must not fall below this baseline." The acceptance threshold is also a regression-test gate: a change that falls below the threshold is not shipped, however attractive it looks.

How often should the golden question set be updated?

The golden question set is a living asset. It is updated on at least two triggers: when a new failure type is seen in production (that example is added to the set) and when the application's scope changes (a new task or user type is added). Beyond that, the set should be reviewed at regular intervals, expected answers that are no longer valid updated, and checked whether it still matches the real usage distribution. The discipline of keeping the set current is the only way an LLM evaluation system preserves its value over time; a set that is not updated falls behind reality and gives a false sense of confidence.

In short, setting up an eval set moves an LLM application's quality from intuition to measurement: you build a golden question set covering real user questions, hard and edge cases, and known failures, tie a rubric to each question and an acceptance threshold to each dimension, calibrate automatic scoring with human evaluator agreement, and run the set like a regression test on every version. A well-built set turns the "it works well" impression into a repeatable and defensible LLM evaluation practice.

This article is a narrow, practical slice of the topic; to see all dimensions including metrics, LLM-as-a-judge, and production monitoring read the comprehensive guide, and for the basics see what is an LLM and what is LLM evaluation. To give your team production-level competency in eval set setup, golden question set design, and acceptance-threshold management, review the corporate training program; a well-built evaluation discipline is the quiet but most decisive assurance of your AI investment.