Prompt & model experiments

Pick a dataset of representative inputs with reference outputs, define evaluators (code checks plus LLM judges with rubrics), run the baseline prompt and the candidate as separate experiments with identical settings and several repetitions, compare scores and per-example differences in the comparison view, review surprising cases manually, then promote the winner by tagging its prompt version.

Start from what users need: correctness against references, format validity, faithfulness to sources, helpfulness, tone and safety, plus cost and latency. Pick a primary metric for the decision and guardrail metrics that must not regress.

Use a fixed dataset that was not used to write the prompt, repeat runs, look at variance and statistical significance, check per-example results instead of averages only, keep a holdout set, and validate wins with real traffic.

Run both on the same representative dataset, score them with evaluators that reflect what matters (correctness, format, tone), compare averages and look at examples where they differ, and check the difference is larger than run-to-run noise. Then check cost and latency before switching.

A judge (person or model) sees two answers to the same input and picks the better one. It is useful for subjective qualities such as helpfulness or tone, where absolute scores are unreliable but comparisons are consistent.

Use the same prompt (adapted only where necessary), dataset, evaluators and parameters, compare quality alongside cost and latency per request, and test on your own data rather than public leaderboards.

Store prompts in a prompt hub or repository with commit history, use tags such as staging and production, link each experiment to a prompt version, and roll back by moving the tag.

When quality is subjective and absolute scores are noisy, such as writing quality or helpfulness; judges (human or LLM) are better at choosing between two outputs than rating one.

Use low temperature for deterministic tasks, run several repetitions for others, fix seeds where providers support them, and report averages with confidence intervals.

Keep tagged slices in the dataset (long inputs, other languages, adversarial inputs, empty fields) and require that each slice's score does not drop, not just the overall average.

Iterate quickly on a few examples in the playground, then run the full dataset as an experiment before deciding; the playground is for exploration, experiments are for decisions.

LangSmith records tokens and latency per run; compare totals and percentiles per experiment and include them in the decision alongside quality.

Use the experiment configuration options for repetitions, concurrency and caching: repetitions run each example several times to measure variance, concurrency runs examples in parallel, and caching reuses results so unchanged parts are not recomputed.

Sourced from Docs by LangChain (MCP)

Run both prompts as experiments on the same dataset, select exactly those two experiments and add them to a pairwise annotation queue. LangSmith pairs runs from the two experiments, and reviewers decide whether run A, run B or both are better for each rubric item.

Sourced from Docs by LangChain (MCP)

Prompts change more often than code and are often edited by people who are not engineers. Versioning them with commits and tags gives history, review and rollback, and lets you promote a tested version without redeploying.

Large enough to detect the differences you care about: tens of examples show big changes, hundreds are needed for small improvements; prioritise quality and coverage of real cases.

Tuning a prompt to pass specific test examples (sometimes by mentioning them) so it scores well on the dataset but generalises poorly to new inputs.

Route a percentage of traffic to the new prompt version, compare online metrics (feedback, task completion, escalations), watch guardrails and stop early if problems appear.

Examples strongly shape outputs; test different example sets as separate variants and make sure examples do not overlap with evaluation data.

Record the hypothesis, prompt versions, dataset version, metrics, results, notable examples and the decision, linked from the change that ships it.

Validate outputs against the schema in code, measure field-level accuracy against references, and check robustness on unusual inputs.

Re-validate the judge against human labels, rerun baseline experiments with the new judge so scores stay comparable, and version judge prompts.

A metric that must not get worse when optimising the primary one, such as safety violations, latency or cost, so improvements do not hide regressions elsewhere.

Datasets can be built from manually curated test cases, historical production traces or synthetic data generation. Mixing real traces with curated edge cases gives a set that reflects both common and difficult inputs.

Sourced from Docs by LangChain (MCP)

Human review, code evaluators for rule-based checks, LLM-as-judge evaluators and pairwise comparisons. Use code checks for objective properties like format, and judges or humans for qualities such as helpfulness.

Sourced from Docs by LangChain (MCP)

Configure online evaluators on the production project with a sampling rate, so a share of live runs is scored automatically, and add failing traces back to the dataset for the next offline experiment.

Sourced from Docs by LangChain (MCP)

Repeat runs to measure variance, look at per-example differences rather than only averages, check the result holds across dataset slices, and treat a difference smaller than run-to-run noise as no difference.

On every prompt or model change, automatically in CI for important prompts, and periodically to detect drift in provider models.

Changes in a provider's model behaviour over time or between versions; pin model versions where possible and rerun experiments when upgrading.

Engineers plus domain experts or product owners, who can judge whether score changes reflect what users value.

Link experiment comparisons in pull requests, summarise wins and trade-offs, and keep a changelog of production prompt versions.

Add examples at and near the context limit to the dataset, measure truncation, latency and whether key instructions are still followed, and compare strategies such as summarising or chunking long inputs.

1. AWhichever looks best
2. BPrompt length
3. CToken count only
4. DA primary metric with guardrailsCorrect answer

Why: Guardrails must not regress.

1. AIt is forbidden
2. BIt uses more tokens
3. CIt overfits the prompt to those examplesCorrect answer
4. DIt is slower

Why: Keep a separate holdout set.

1. ARestart servers
2. BMove the production tag to its versionCorrect answer
3. CCopy-paste into code by hand
4. DEmail it

Why: Rollback by moving the tag back.

1. AUse the longest prompt
2. BTest each on different questions
3. CRun both on the same datasetCorrect answer
4. DOnly test the newer one

Why: The same inputs make the scores comparable.

1. AFormat checks
2. BLatency
3. CSubjective quality like writingCorrect answer
4. DExact numeric answers

Why: Choosing is easier than rating.

1. ASame dataset, evaluators and parametersCorrect answer
2. BDifferent datasets
3. CPublic leaderboards only
4. DOne example

Why: Plus cost and latency.

1. AChecking edge-case groups do not regressCorrect answer
2. BMaking datasets smaller
3. CSpeeding up runs
4. DHiding failures

Why: Averages hide slice drops.

1. ABilling
2. BQuick exploration before full experimentsCorrect answer
3. CFinal decisions
4. DProduction traffic

Why: Full experiments make the decision.

1. APrompt length
2. BFont size
3. CThings like safety, cost and latencyCorrect answer
4. DOnly accuracy

Why: No hidden regressions.

1. ARepetitions with confidence intervalsCorrect answer
2. BOne run only
3. CHigher temperature
4. DIgnoring it

Why: Results vary between runs.

1. ARepetitionsCorrect answer
2. BCaching
3. CA smaller dataset
4. DTurning off evaluators

Why: Running each example several times shows the noise.

Sourced from Docs by LangChain (MCP)

1. AOne experiment
2. BFive prompts
3. CNo dataset
4. DExactly two experiments on the same datasetCorrect answer

Why: Runs from the two experiments are paired for review.

Sourced from Docs by LangChain (MCP)

1. AA new prompt
2. BA token count only
3. CA SQL query
4. DWhich of two answers is betterCorrect answer

Why: It compares two outputs for the same input.

1. AIt is cheaper to log
2. BSo you know which change caused the differenceCorrect answer
3. CLangSmith requires it
4. DModels refuse multiple changes

Why: Changing the prompt and the model together hides which one helped.

1. AUsing examples
2. BScoring well on tests but generalising poorlyCorrect answer
3. CPrompts that are too long
4. DToo many tokens

Why: Validate on new inputs.

1. AHundreds of examplesCorrect answer
2. BTwo examples
3. CTen words
4. DOne example

Why: Big changes need fewer.

1. AA press release
2. BA gradual production A/B testCorrect answer
3. CDeleting the old prompt
4. DNo monitoring

Why: Watch online metrics.

1. AEvaluation dataCorrect answer
2. BThe system prompt
3. CThe model name
4. DLogs

Why: Otherwise results are inflated.

1. ABy length
2. BBy speed only
3. CSchema validation and field accuracyCorrect answer
4. DBy eye only

Why: Run code checks before judges.

1. ACompare with old scores directly
2. BDelete experiments
3. CIgnore it
4. DRe-validate it and rerun baselinesCorrect answer

Why: Keep scores comparable.

1. AHypothesis, versions, metrics and decisionCorrect answer
2. BOnly the winner
3. CNothing
4. DThe model's weights

Why: Linked from the shipping change.

1. AGit
2. Bnpm
3. CThe browser
4. DLangSmith tracingCorrect answer

Why: Compare per experiment.

1. ARandom internet text
2. BThe model's own outputs only
3. CCurated cases, production traces or synthetic dataCorrect answer
4. DOnly the developer's memory

Why: Mixed sources cover common and edge cases.

Sourced from Docs by LangChain (MCP)

1. AOnly the dataset
2. BEvery prompt draft
3. COld experiments
4. DA sampled share of production runsCorrect answer

Why: Sampling keeps monitoring affordable.

Sourced from Docs by LangChain (MCP)

1. AReuses results so unchanged work is not recomputedCorrect answer
2. BHides failures
3. CChanges the prompt
4. DDeletes datasets

Why: It saves time and cost on repeat runs.

Sourced from Docs by LangChain (MCP)

1. ANo real differenceCorrect answer
2. BA clear winner
3. CThe dataset is perfect
4. DThe judge is broken

Why: Noise can create apparent gaps.

1. ADisabling evaluation
2. BPinning versions and rerunning experimentsCorrect answer
3. CIgnoring updates
4. DUsing random models

Why: Providers change models.

1. AEngineers with domain expertsCorrect answer
2. BOnly the model
3. CNobody
4. DOnly finance

Why: Scores must match user value.

1. ANever
2. BOnly after outages
3. COn every change, automatically in CICorrect answer
4. DOnce a year

Why: Catch regressions early.

1. AServer logs
2. BExperiment comparisonsCorrect answer
3. CMemes
4. DNothing

Why: Reviewers see the evidence.

1. ACreative brainstorming only
2. BImage generation
3. CNothing
4. DDeterministic tasks like extractionCorrect answer

Why: Less variation between runs.

1. ATruncation, latency and instruction followingCorrect answer
2. BFont rendering
3. CUpload speed
4. DDisk space

Why: Long inputs often break instructions or hit limits.