Curating golden datasets from production

I'd start by tracing everything with useful metadata, then run sampled online evaluators to flag likely failures. Automation rules send flagged runs and negative feedback to an annotation queue, and a stratified sample of normal traffic to curation. Humans correct the reference outputs, because the production answer may be wrong. A curation step scrubs PII, dedupes by embedding similarity and records the source trace. Examples go into splits by purpose, regression, representative, edge-case and held-out, and each change is tagged as a new version. Experiments with evaluate() always pin a version. Finally a scheduled job compares traffic clusters with dataset coverage to add new intents and retire stale examples, so the set keeps reflecting reality.

Sourced from Docs by LangChain (MCP)

First, I instrument the app with LANGSMITH_TRACING plus @traceable or wrap_openai, or OpenTelemetry through the OTLP endpoint, and tag runs with feature, prompt version and user segment. Online evaluators on the production project score a filtered sample and mark candidate failures. Automation rules send low scores, negative feedback and errors to a single-run annotation queue, and a small random or stratified sample of normal traffic goes to curation. Reviewers fix reference outputs and export accepted runs. A curation job scrubs PII, normalises inputs, deduplicates and records lineage. Synthetic variants are generated from reviewed failures and reviewed too. Examples go into named splits, and the version is tagged. evaluate() and pytest run against that pinned version, and a freshness job keeps coverage current.

Sourced from Docs by LangChain (MCP)

I treat it as a hard invariant enforced in the pipeline, not a convention. The only path from a failing or flagged trace into the dataset goes through a single-run annotation queue, where a reviewer applies the rubric, corrects the output and explicitly accepts it. Representative samples and synthetic variants also need a review step before they are written. I record who reviewed each example and when in its metadata, and the curation job refuses to write any example that lacks that field. I never copy the model's raw output straight into the reference field. Writing examples with create_examples from a single controlled job, rather than letting people add them ad hoc, makes this easy to audit.

Sourced from Docs by LangChain (MCP)

I scrub PII in the curation job before anything is written with create_examples, so it covers queue exports, representative samples and synthetic data alike. The scrubber detects things like names, emails, phone numbers, account IDs and addresses in both inputs and reference outputs, and replaces them with consistent placeholders so the example still makes sense. I also tell reviewers to flag PII the scrubber missed, and I check that synthetic generators do not invent realistic personal data. Raw production traces still exist in the tracing project under its own access controls and retention, but the dataset itself, which is copied around, shared and used in CI, holds only scrubbed content. If scrubbing fails, the example is rejected, not written.

Sourced from Docs by LangChain (MCP)

An experiment score is a function of both the app and the examples. If the dataset changes between two runs, a drop could just mean we added harder cases. Pinning a tagged version means both experiments see identical examples, so the delta is about the app. Splits matter because each answers a different question. The regression split asks whether we broke something we already fixed, and it gates CI. The representative split estimates real-world quality. The edge-case and synthetic split stresses robustness. The held-out split stays untouched during prompt tuning so we have an honest final check. Mixing them hides which kind of quality moved.

Sourced from Docs by LangChain (MCP)

I use synthetic data where production is thin: rare intents, adversarial inputs, long contexts, new languages, or a feature not yet launched. The best seeds are real reviewed failures, so variants test the same weakness from different angles. To stop distortion, every synthetic example is tagged by source and lives in its own split, and I report its scores separately from real traffic. A human reviews generated examples before they enter the dataset, because generators produce unrealistic or wrongly labelled cases. I also cap how many variants one seed can create, otherwise a single bug dominates the aggregate. Once real traffic covers an area, I prefer real examples and retire synthetic ones.

Sourced from Docs by LangChain (MCP)

I tag each run with the feature or entry point, the prompt version or Prompt Hub commit, the model, the user segment, and for chat flows the thread ID. Curation depends on filtering. Online evaluators need filters so they score the right traffic. Automation rules route by feature or segment. Stratified sampling needs a segment field to stratify on. Later, when the freshness job retires examples tied to a removed feature, it can only do that if the feature was recorded. Prompt version metadata lets me tell whether a failure belongs to an old prompt that is already fixed. Adding metadata later is painful because historic traces will not have it, so I decide on the schema before launch.

Sourced from Docs by LangChain (MCP)

I filter to the traffic I actually want to curate, such as a specific feature, production environment and root runs, and leave out internal test users. The sampling rate is a trade-off between coverage and cost. LLM-as-judge evaluators cost tokens and add latency to feedback, and every scored trace is upgraded to extended retention, which also costs money. Cheap code rules, like format or schema checks, can run at a higher rate than judges. I raise sampling for new or high-risk features and lower it for stable, high-volume ones. Errors and negative user feedback do not need sampling because automation rules can route them directly. I review the rates whenever queue volume gets ahead of reviewer capacity.

Sourced from Docs by LangChain (MCP)

A dataset made only of failures is skewed. It measures how well you fix past bugs, not how well the app serves typical users, and a change can pass every regression case while quietly breaking common requests. Representative examples protect the core experience and give scores that look roughly like production quality. Failures and synthetic edge cases protect the tails. Keeping them in separate splits lets me report each one on its own, so a gain on edge cases cannot hide a drop on representative traffic. The representative sample is small and random or stratified, and its reference outputs are still reviewed. Being normal traffic does not mean the model's answer was correct.

Sourced from Docs by LangChain (MCP)

Each example has inputs that match the app's entry-point schema, so evaluate() can call the target directly, and reference outputs corrected by a human. The metadata holds the source, which is production, synthetic or manual, plus the source trace ID for lineage, the intent cluster, the feature, the date added, the reviewer and review date, and for synthetic examples the seed example it came from. Each example also belongs to a split: regression, representative, edge-case or held-out. For agents I might add an expected trajectory or tool calls, and for conversations the prior turns from the thread. The dataset has versions and tags, so any experiment can name the exact snapshot it ran against.

Sourced from Docs by LangChain (MCP)

I use them for triage, not as ground truth. A low judge score only nominates a trace for human review. It never decides the reference output on its own. Judges have false positives and false negatives and can be biased towards length or style, so I calibrate them. I compare judge verdicts with reviewer decisions from the queue, and I adjust the prompt, the few-shot examples or the threshold when they drift apart. Corrected human labels can become few-shot examples for the judge. Because low scores alone miss some failures, I also route on negative user feedback, errors and cheap code rules. The rate of false alarms reviewers see is a useful health metric.

Sourced from Docs by LangChain (MCP)

I use regression for reviewed past failures that must stay fixed, representative for sampled normal traffic, edge-case for synthetic and adversarial variants, and held-out for a protected set not used during iteration. Regression gates merges through the pytest @pytest.mark.langsmith suite, because it is deterministic in intent and directly tied to known bugs. It should also be small enough to run on every pull request. Representative and edge-case run in fuller evaluate() experiments before release and are compared with the baseline. Held-out runs rarely, as a final check. Every split is read from the same pinned dataset version, so the CI gate and the release experiments describe the same data.

Sourced from Docs by LangChain (MCP)

When engineers tune prompts against a dataset repeatedly, they overfit to it. Scores climb while real quality stays flat. A held-out split gives an honest estimate because nobody optimises against it. To protect it, I limit who looks at individual examples, run it only at milestones such as release candidates, and never use its examples as few-shot examples for prompts or judges. I also keep its items out of synthetic seeds, or the variants would leak into other splits. If it gets contaminated, for example by being used in day-to-day debugging, I add fresh reviewed examples, retire the leaked ones in a new version, and note that in the changelog.

Sourced from Docs by LangChain (MCP)

Tests marked @pytest.mark.langsmith run the target on the regression split. LangSmith records them as a dataset and an experiment, so every CI run can be compared with the baseline. The test loads examples from a pinned dataset tag, so CI does not change when someone adds examples. For nondeterminism, I prefer assertions on properties, like format, required facts or tool choice, over exact string match. I use evaluators with tolerances, and I can rely on repetitions to see variance. I gate on clear thresholds or 'no regression on previously passing examples' instead of single flaky asserts. Caching the target or judge calls for unchanged inputs keeps CI fast and cheap.

Sourced from Docs by LangChain (MCP)

On a schedule, it clusters recent production traces by intent and compares the clusters with the intent clusters recorded in dataset metadata. Intents that are common in traffic but missing or thin in the dataset are sent to the annotation queue, so humans curate them like any other candidate. Examples tied to features that have been removed or changed are retired. The job never edits the existing dataset. It writes the changes and tags a new version, so older experiments stay reproducible. It also produces a short report of coverage gaps, retirements and additions, so whoever owns evaluation can see the dataset tracking the product and spot surprises such as a sudden new intent.

Sourced from Docs by LangChain (MCP)

Experiments are only comparable if they ran on the same data. If I edited the current dataset in place, last week's baseline and today's run would quietly refer to different contents, and an old result could no longer be reproduced. Tagging a new version, for example moving from 'prod-2026-w10' to the next weekly tag, keeps old snapshots intact. When a team moves to the new version, it reruns the baseline against it so the comparison is fair. This also gives an audit trail of what was retired and why. If a retirement turns out to be wrong, I can restore from the earlier version.

Sourced from Docs by LangChain (MCP)

Even scrubbed, a production-derived dataset is sensitive, so I separate permissions. Few people can read raw production tracing projects. Reviewers can see only the annotation queues they work. CI and engineers get read access to the dataset, and only the curation job's service identity can write to it. API keys stay in a secrets manager and are scoped per environment. I keep source trace IDs so that when a customer asks for their data to be deleted, I can find derived examples and retire them in a new version. Where policy requires it, I may also purge the content from old versions. Synthetic generation and LLM-as-judge calls must follow the same data-processing rules as production.

Sourced from Docs by LangChain (MCP)

It depends on how skewed the traffic is. Pure random sampling mirrors production frequencies, which is good for estimating overall quality, but a few dominant intents swamp it and small segments may never appear. Stratified sampling by feature, intent cluster or user segment makes sure low-volume but important segments, such as a new language or an enterprise tier, are covered. My usual approach is stratified sampling with a minimum per stratum, while recording the stratum in metadata. If I need a production-weighted number, I can reweight using the known traffic mix. Whatever I choose, I document it in the dataset description so people reading the scores know what they represent.

Sourced from Docs by LangChain (MCP)

Production traces store inputs at whatever level was traced. That might be a nested LLM call, an already-formatted prompt, or extra framework fields. If I copied those as-is, the evaluation would test a fragment of the system, or tie the dataset to an old prompt template. Normalising to the entry-point schema, meaning the same arguments a user request passes to the top-level function, lets evaluate() run the whole current pipeline on every example. It also lets old and new examples sit together after internal refactors. I validate each example against that schema in the curation job and reject malformed ones. When the entry-point schema changes, that becomes a new dataset version.

Sourced from Docs by LangChain (MCP)

I embed the normalised inputs and compare each candidate with existing examples and with the other candidates. Anything above a similarity threshold counts as a near-duplicate, and I keep one, preferring the reviewed failure or the most recent one. This stops one popular query from dominating scores. The risk is the threshold. If it is too loose, inputs that look alike but need different answers get merged, such as the same question about two different products, or a negation. If it is too strict, duplicates get through. I tune it by checking merged pairs by hand, apply it within an intent cluster, and log the IDs of removed examples so merges can be undone.

Sourced from Docs by LangChain (MCP)

A reservation locks an item to one reviewer for a while, so two people do not correct the same trace and produce conflicting reference outputs or waste effort. If the reviewer drops it, the reservation lapses and the item goes back to the queue. The reviewer count sets how many people must review each item. For most failures one careful reviewer is enough. For high-stakes or ambiguous domains I ask for more, so I can measure agreement and escalate disagreements. Together they let a small team work a shared queue safely. They also give a clear record of who approved each reference answer, which supports the rule that nothing unreviewed reaches the dataset.

Sourced from Docs by LangChain (MCP)

The rubric has to support two decisions: whether the trace belongs in the dataset, and what the correct reference output is. I include criteria such as factual correctness against the source, following instructions, format, safety and tone. I add assertions that are easy to check, like 'cites a document' or 'does not promise refunds'. I also add explicit reject reasons: duplicate, out-of-scope feature, PII-heavy, ambiguous input, or a bug in the app rather than the model. Reviewers should write the reference to the standard we want, not only lightly edit the model's answer. A short guide with worked examples cuts disagreement, and I revise it whenever disputes keep coming back.

Sourced from Docs by LangChain (MCP)

I do not write either version yet. The item goes to an adjudicator, usually a domain owner, who chooses one or writes a new reference. Disagreement often shows that the input is ambiguous or the rubric is unclear. If the input is genuinely ambiguous, I might keep it with a reference that allows several valid answers, or score it with a rubric-based evaluator instead of exact match. Sometimes I reject it. I track the disagreement rate per intent cluster, because repeated conflict means the guide needs work. For subjective qualities, pairwise comparison, or a human or LLM-as-judge evaluator, may suit better than one fixed golden answer.

Sourced from Docs by LangChain (MCP)

I do it as arithmetic with our own numbers, not assumed ones. Flagged volume per day is production traces, times the online evaluator sampling rate, times the share that score low, plus negative-feedback and error traces, plus the representative sample. Capacity is reviewers, times focused hours, times items per hour, divided by reviewer count per item. If inflow exceeds capacity, the backlog grows and traces may expire unless retention was extended. My levers are lowering sampling rates, routing only the most severe or novel failures, deduplicating before queueing so near-identical failures arrive once, and prioritising uncovered intents. I watch queue age as the key metric.

Sourced from Docs by LangChain (MCP)

I make writes idempotent. Each example gets a deterministic key, such as the source trace ID plus a hash of the normalised inputs, stored in metadata. Before calling create_examples, the job checks which keys already exist and writes only the missing ones in batches. The new version is tagged only after all writes and validations succeed, so experiments pinned to a tag never see a half-written state. Failed items, such as PII scrubbing errors or schema problems, go to a dead-letter list with the reason instead of being dropped silently. Because the job reads accepted exports and not raw traffic, a rerun gives the same result.

Sourced from Docs by LangChain (MCP)

Curation is not instant. A failure might sit in a queue for days before a reviewer gets to it. Running an online evaluator on a trace upgrades it to extended retention, so flagged candidates stay available while they wait. Traces that were never scored stay on base retention and could expire before anyone looks at them, which matters for traces routed by user feedback or errors alone. To guard against that, I make sure those routes also give extended retention, or I export the needed content into the queue promptly. I also keep queue age well within the retention window. If a trace has expired, the candidate is lost unless the failure shows up again.

Sourced from Docs by LangChain (MCP)

LangSmith groups turns into threads, so I curate at the thread level when the failure depends on context. The example input carries the prior conversation history up to the failing turn, in the entry-point message format, and the reference output is the corrected reply for that turn. When quality is about the whole conversation, such as whether the issue was resolved, online evaluators can run on threads, and I can store a thread-level rubric instead of one reference reply. I scrub PII across every turn, not only the last one, because personal details often appear early in a chat. Deduplication needs care, since conversations with the same opening can still differ later.

Sourced from Docs by LangChain (MCP)

For agents, a correct final answer reached through wrong or unsafe tool calls is still a failure. So I store an expected trajectory, or the key tool calls, alongside the reference output, taken from the reviewed trace and corrected by the reviewer. agentevals provides trajectory evaluators, including a trajectory LLM-as-judge, which can score whether the agent took a reasonable path without needing an exact match. I usually check strict criteria, such as never calling a dangerous tool, with code rules, and leave looser judgments to the judge. Inputs still match the entry point, so evaluate() runs the whole agent. External tools are mocked or sandboxed to keep runs repeatable.

Sourced from Docs by LangChain (MCP)

If reviewers lightly edit the model's answers, the references take on its phrasing and blind spots, and future models are rewarded for imitating it. To avoid this, I tell reviewers to write the reference to the target standard, using source documents and policy, and to judge the model's answer instead of polishing it. I prefer evaluators that check substance, like required facts, rubric criteria or tool choice, over string similarity to a single reference. For synthetic data, I avoid generating references with the same model under test, and I still review them. Sampling uncovered intents, not only failures of the current model, also helps.

Sourced from Docs by LangChain (MCP)

Prompts are managed in the Prompt Hub with commits and tags. The target in an evaluate() run, or in a pytest test, pulls a specific commit or tag instead of 'latest'. I record the prompt commit, the model and the dataset version tag in the experiment metadata, so a comparison states exactly what changed. In production, the prompt version is tagged on runs at instrumentation time. That lets curation tell whether a failure came from a prompt that has already been replaced. An experiment is then fully described by three pins: the dataset version and split, the prompt commit, and the model and settings.

Sourced from Docs by LangChain (MCP)

I check that it predicts production. Releases that score better on it should also improve online evaluator scores and user feedback afterwards. If the dataset passes changes that then fail in production, it is missing coverage. Those escaped failures go into the regression split, and the freshness job closes the intent gap. I also track coverage of current intent clusters, the age of examples, reviewer agreement, the mix of sources, and whether scores have plateaued near the ceiling, which suggests the cases have become too easy. A good dataset should also catch known bugs on purpose: if I bring back a fixed issue, the regression split should flag it.

Sourced from Docs by LangChain (MCP)

If the service is already instrumented with OpenTelemetry, or written in a language or framework where an OTel setup is the natural choice, I send spans to LangSmith's OTLP endpoint instead of adding a second tracing system. When the same traces also have to reach an existing observability backend, an OpenTelemetry collector can fan them out to both. The SDK route, with LANGSMITH_TRACING and @traceable or wrap_openai, is simpler for Python and JS LLM code and fits LangChain and LangGraph directly. Either way, the curation pipeline needs the same metadata attributes, such as feature, prompt version and segment, so I standardise those names across both paths.

Sourced from Docs by LangChain (MCP)

Backtesting runs a candidate version on inputs that production has already seen, to check how it would have done. The representative split, built from real traces with human-corrected references, fits well. I run evaluate() for the candidate and for the current production version against the same pinned version and split, then compare the experiments example by example. Recent traffic not yet curated can also be replayed, with reference-free evaluators or pairwise comparison, since there may be no reference answer. Interesting differences can go to a pairwise annotation queue comparing the two experiments. That way the decision rests on real usage patterns as well as synthetic ones.

Sourced from Docs by LangChain (MCP)

1. AOnly by the online LLM-as-judge
2. BOnly in the evaluate() output
3. CIn the curation job before examples are writtenCorrect answer
4. DAfter the dataset version is tagged

Why: Step 5 scrubs PII in the curation job, before any example becomes part of the dataset.

Sourced from Docs by LangChain (MCP)

1. AIt is deleted after evaluation
2. BIt is upgraded to extended retentionCorrect answer
3. CIt is copied to every dataset automatically
4. DIts retention is unchanged

Why: Running an online evaluator on a trace upgrades it to extended retention, which keeps flagged traces available for later curation.

Sourced from Docs by LangChain (MCP)

1. ARun evaluate() against a pinned, tagged dataset versionCorrect answer
2. BAdd new failures directly from production outputs
3. CIncrease the online evaluator sampling rate
4. DAlways run against the latest dataset

Why: Pinning a tagged version keeps the examples identical across experiments, so differences come from the application under test.

Sourced from Docs by LangChain (MCP)

1. AOTEL_SDK_DISABLED
2. BLANGSMITH_DATASET
3. CLANGSMITH_TRACINGCorrect answer
4. DLANGCHAIN_DEBUG

Why: Setting LANGSMITH_TRACING turns on tracing, used together with @traceable or wrappers such as wrap_openai.

Sourced from Docs by LangChain (MCP)

1. A@dataset
2. B@pytest.mark.langsmith
3. C@traceableCorrect answer
4. D@evaluator

Why: @traceable records a function call as a run in LangSmith; the pytest marker is for tests.

Sourced from Docs by LangChain (MCP)

1. AExtended retention
2. BSampling rates
3. CA reservationCorrect answer
4. DA dataset split

Why: Annotation queues support reservations, which lock an item to one reviewer.

Sourced from Docs by LangChain (MCP)

1. A@pytest.mark.slow
2. B@pytest.mark.parametrize
3. C@traceable
4. D@pytest.mark.langsmithCorrect answer

Why: The LangSmith pytest integration uses @pytest.mark.langsmith.

Sourced from Docs by LangChain (MCP)

1. AMixed in unlabelled so they look like production data
2. BLabelled source=synthetic and reviewed before enteringCorrect answer
3. CWritten directly with model-generated references
4. DPlaced only in the held-out split without review

Why: Synthetic variants are labelled source=synthetic and reviewed, so they never become unreviewed references.

Sourced from Docs by LangChain (MCP)

1. AMove them to the regression split
2. BWrite the changes and tag a new dataset versionCorrect answer
3. CDelete them from the current version in place
4. DRename the dataset

Why: The workflow tags a new version instead of editing the old one, so past experiments stay reproducible.

Sourced from Docs by LangChain (MCP)

1. ARegressionCorrect answer
2. BRepresentative
3. CHeld-out
4. DEdge-case

Why: Step 8 gates merges on the regression split through @pytest.mark.langsmith tests.

Sourced from Docs by LangChain (MCP)

1. ALabelling synthetic examples
2. BExporting a flagged trace's raw model output as the reference without reviewCorrect answer
3. CTagging a new version after retirements
4. DPinning a dataset tag in pytest

Why: Unreviewed outputs must never become reference answers.

Sourced from Docs by LangChain (MCP)

1. AOne
2. BExactly twoCorrect answer
3. CUp to five
4. DAny number

Why: Pairwise queues are designed for exactly two experiments, letting reviewers pick the better of two outputs for the same input.

Sourced from Docs by LangChain (MCP)

1. AWrap the client with wrap_openaiCorrect answer
2. BAdd the client to an annotation queue
3. CExport a CSV of calls each night
4. DUse create_examples on each response

Why: wrap_openai wraps the OpenAI client so its calls are traced automatically.

Sourced from Docs by LangChain (MCP)

1. ARun two separate apps
2. BExport datasets to the other backend
3. CSend OpenTelemetry spans through a collector that fans out to bothCorrect answer
4. DDuplicate every @traceable call

Why: LangSmith supports native OpenTelemetry via an OTLP endpoint, and a collector can fan traces out to several backends.

Sourced from Docs by LangChain (MCP)

1. APrompt tags
2. BThreadsCorrect answer
3. CSplits
4. DDatasets

Why: Runs and traces are grouped into projects, and threads group multi-turn conversations.

Sourced from Docs by LangChain (MCP)

1. AExperiment queue
2. BPairwise queue
3. CThread queue
4. DSingle-run queueCorrect answer

Why: Single-run queues provide rubrics, assertions and export to datasets; pairwise queues compare exactly two experiments.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai
2. BPrompt Hub
3. COTLP collector
4. DagentevalsCorrect answer

Why: agentevals provides trajectory evaluators, including a trajectory LLM-as-judge.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai
2. Btraceable
3. Cevaluate()
4. Dcreate_examplesCorrect answer

Why: Step 7 writes examples with create_examples into splits such as regression and held-out.

Sourced from Docs by LangChain (MCP)

1. Acreate_examples
2. Bwrap_openai
3. Cevaluate()Correct answer
4. Dtraceable

Why: Experiments run through evaluate(), which supports repetitions, concurrency and caching.

Sourced from Docs by LangChain (MCP)

1. AStraight into the dataset
2. BTo the Prompt Hub
3. CTo be deleted
4. DTo an annotation queueCorrect answer

Why: Candidate failures go to an annotation queue for human review; only a small sample of normal traffic goes straight to curation.

Sourced from Docs by LangChain (MCP)

1. ABy prompt tag
2. BBy embedding similarityCorrect answer
3. CBy exact trace ID match only
4. DBy the reviewer's name

Why: Step 5 deduplicates near-identical examples using embedding similarity.

Sourced from Docs by LangChain (MCP)

1. ARepetitions
2. BPairwise queues
3. CDataset splits
4. DA filter and a sampling rateCorrect answer

Why: Online evaluators run on production runs or threads with filters and sampling rates.

Sourced from Docs by LangChain (MCP)

1. AIt sets the sampling rate
2. BIt speeds up the LLM-as-judge
3. CIt replaces the reference output
4. DIt provides lineage back to the original production runCorrect answer

Why: The source trace ID links an example to its trace, which helps with audits, debugging and finding derived data.

Sourced from Docs by LangChain (MCP)

1. ATo measure variance from nondeterministic LLM outputsCorrect answer
2. BTo skip human review
3. CTo extend trace retention
4. DTo make the dataset larger

Why: Repetitions run each example several times, so a score change can be told apart from run-to-run noise.

Sourced from Docs by LangChain (MCP)

1. ATo the annotation queueCorrect answer
2. BDirectly into the held-out split
3. CTo the Prompt Hub
4. DNowhere; it only logs them

Why: Step 9 sends uncovered intents to the queue so humans curate them.

Sourced from Docs by LangChain (MCP)

1. AIt guarantees zero PII
2. BIt replaces online evaluators
3. CIt removes the need for review
4. DIt ensures smaller but important segments are coveredCorrect answer

Why: Stratifying by feature, intent or segment stops dominant traffic from crowding out small segments.

Sourced from Docs by LangChain (MCP)

1. ASo evaluate() can run the full current pipeline on each exampleCorrect answer
2. BTo create pairwise queues
3. CTo shrink trace storage
4. DTo bypass annotation queues

Why: Entry-point inputs let the target be called end to end, regardless of how the inner steps were traced.

Sourced from Docs by LangChain (MCP)

1. ATo store retired examples
2. BTo hold PII safely
3. CTo give an unbiased check that is not used for day-to-day tuningCorrect answer
4. DTo gate every pull request

Why: A held-out split detects overfitting to the data that teams iterate on.

Sourced from Docs by LangChain (MCP)

1. AUnreviewed production outputs
2. BRetired examples
3. CRandom strings
4. DReviewed failuresCorrect answer

Why: Step 6 uses reviewed failures as seeds for paraphrases, long, adversarial and multilingual variants.

Sourced from Docs by LangChain (MCP)

1. ABy dataset splits
2. BBy annotation reservations
3. CBy OTLP spans
4. DWith commits and tags in the Prompt HubCorrect answer

Why: Prompts are versioned and managed in the Prompt Hub / Playground with commits and tags.

Sourced from Docs by LangChain (MCP)

1. ACode rule evaluator
2. BFormat check
3. CRetention rule
4. DPairwise evaluatorCorrect answer

Why: LangSmith supports pairwise evaluators for comparing two outputs or experiments.

Sourced from Docs by LangChain (MCP)

1. AIt avoids repeating identical calls, saving time and costCorrect answer
2. BIt assigns reviewers
3. CIt scrubs PII
4. DIt versions the dataset

Why: Caching reuses results for identical calls, which makes repeated experiments and CI cheaper and faster.

Sourced from Docs by LangChain (MCP)

1. ATurn test files into datasets and experimentsCorrect answer
2. BReplace online evaluators
3. CManage annotation reservations
4. DScrub PII from traces

Why: Like the pytest integration, Vitest/Jest integrations turn test files into datasets and experiments.

Sourced from Docs by LangChain (MCP)