LangSmith blueprint · AI / LLM engineering

Calibrating LLM-as-judge evaluators

Builds an LLM-as-judge prompt with few-shot examples, measures its agreement with human labels collected in a LangSmith annotation queue, and iterates until it matches reviewers before it is trusted in production. It must never report judge quality using examples the judge has already seen as few-shots, and never ship a judge that has not been checked against humans on each important slice.

Steps: 9
Subjective questions: 33
Objective questions: 33

The problem and the outcome

Problem: Teams use LLM judges to score thousands of runs, but an uncalibrated judge can quietly disagree with domain experts. Every dashboard, regression test and alert built on it then measures the wrong thing.

Outcome: The judge prompt is a versioned Prompt Hub commit with a measured, per-slice agreement score against human reviewers. Production disagreements flow back into the annotation queue, so calibration is maintained instead of being assumed.

Architecture

How it works, step by step

Sample production runs from the tracing project, deliberately including failures and edge cases, into a single-run annotation queue. Set a written rubric and assign at least two reviewers per item so you can measure human agreement.
Resolve reviewer disagreements, then export the labeled runs to a calibration dataset, with the human verdict as the reference output. Split it into a small few-shot pool and a larger held-out evaluation split.
Write the judge prompt in the Prompt Hub/Playground, starting from a prebuilt LLM-as-judge template if one fits. Add the rubric, a structured output (verdict plus reasoning) and few-shot examples drawn only from the few-shot pool, then commit it.
Run evaluate() with the pinned judge commit over the held-out split, using repetitions to expose run-to-run variance. Each judge verdict is recorded as feedback on the experiment.
A row-level code evaluator checks whether the judge verdict matches the human label. A summary evaluator computes Cohen's kappa and per-class precision and recall across the whole experiment.
Compare the agreement with the human-to-human agreement ceiling and the bar you agreed with reviewers. Check overall agreement and each important slice, such as language, intent or the rare 'fail' class.
If the judge misses the bar, open the disagreements side by side across experiments, find the pattern (vague criterion, missing edge case, verbosity bias), and revise the rubric or add targeted few-shots. Commit a new version and rerun on the same held-out split.
When it passes, tag that prompt commit for production. Add the calibration experiment to a pytest regression test (@pytest.mark.langsmith) so future prompt or model changes are re-checked.
Attach the tagged judge as an online evaluator on production runs with a filter and sampling rate, and feed scores into dashboards and alerts. An automation rule sends low-scoring and borderline runs back to the annotation queue, so humans keep spot-checking the judge.

Tech stack

LangSmith annotation queues (single-run, rubric, reviewer counts)
LangSmith datasets with splits
Prompt Hub / Playground with commits and tags
langsmith SDK evaluate() with row and summary evaluators
Judge LLM with structured output
Online evaluators, dashboards and alerts
Automation rules routing runs to queues
pytest integration (@pytest.mark.langsmith) for regression

Key design decisions

Use Cohen's kappa and per-class recall rather than raw accuracy, because real data is usually skewed towards 'pass' and raw accuracy hides a judge that rarely flags failures.
Keep few-shot examples and held-out evaluation examples in separate dataset splits, so the reported agreement is not inflated by the judge grading cases it was shown.
Prefer binary or small categorical verdicts with reasoning over 1–10 scales, since coarse labels are easier for humans to agree on and for the judge to match.
Pin the judge to a specific prompt commit and model version, and treat any change to either as a new judge that must be recalibrated.
Set the target relative to human-to-human agreement; you cannot expect the judge to agree with reviewers more than reviewers agree with each other.

Common mistakes

Reporting agreement on the same examples used as few-shots, which overstates quality.
Calibrating only on easy, clean cases from one source, so the judge drifts badly on real production traffic.
Iterating the prompt dozens of times against one small held-out split until it overfits that split; refresh it with new labels periodically.
Ignoring judge biases such as preference for longer answers or, in pairwise mode, for the first position; swap order and check.
Using a single reviewer per item, which leaves no way to tell judge error from label noise.

What to measure

Cohen's kappa between judge and consensus human label, overall and per slice
Recall and precision of the judge on the minority 'fail' class
Human inter-annotator agreement (the ceiling) on the same items
Judge self-consistency across repetitions of the same example
Agreement on freshly re-labeled production samples over time (drift)

Most-asked interview questions

Subjective questions 33

1How would you prove an LLM-as-judge evaluator is trustworthy before using it in CI?Asked almost every time

I treat the judge like a model that needs its own test set. First I get domain reviewers to label a representative sample in an annotation queue, with two reviewers per item, so I know the human agreement ceiling. I export that to a dataset with separate few-shot and held-out splits. Then I run the pinned judge prompt with evaluate() over the held-out split and compute Cohen's kappa and per-class recall with a summary evaluator, overall and per slice. I only promote it when it is close to human agreement on every slice that matters, especially the failure class. The calibration experiment then becomes a regression test, so any prompt or model change is re-checked automatically.