Evals in CI with pytest and Vitest

I separate hard and soft checks. Hard checks are deterministic, things like a valid JSON schema, the correct tool being called, or no PII in the output, and they fail the build immediately. Soft checks come from LLM-as-judge scores. I never gate a soft check on one sample. I run repetitions or use low temperature, average the scores, and fail only below a threshold I calibrated against human labels, or on a meaningful drop versus the main-branch baseline experiment. In pull requests, the cache replays unchanged calls, so only the parts that changed are re-evaluated. I also track a flake rate. Any test that flips without a code change gets quarantined and fixed rather than retried until it goes green.

Sourced from Docs by LangChain (MCP)

A pull request triggers a CI job that holds LANGSMITH_API_KEY as a secret, a stable LANGSMITH_TEST_SUITE name, and optionally a LANGSMITH_TEST_CACHE directory. The runner executes pytest tests marked @pytest.mark.langsmith, or Vitest files that use ls.describe and ls.test with the LangSmith reporter. Each test case becomes one dataset example. The test calls the real chain or agent, traced with wrap_openai or @traceable, using prompts pulled from the Prompt Hub by tag. Hard rules are asserted. Softer qualities are scored by judges and logged as feedback. The integration upserts the dataset and records the run as a new experiment. Any failed assertion makes the runner exit non-zero, the check turns red, and branch protection blocks the merge. Reviewers compare the experiment against the main-branch baseline, and disputed cases go to annotation queues.

Sourced from Docs by LangChain (MCP)

The test suite maps to a dataset, and the dataset is named by LANGSMITH_TEST_SUITE. Each test case, including each parametrized case, becomes an example. Its inputs and reference outputs come from the test parameters, logged with t.log_inputs and t.log_reference_outputs in pytest or the Vitest equivalents. Every CI run creates a new experiment over that dataset. The experiment holds a run per example with the actual outputs from t.log_outputs, the full trace of the chain or agent, and feedback entries from t.log_feedback or ls.logFeedback. Because the dataset stays the same and experiments accumulate, any two runs can be compared example by example. That comparison is what makes regression review possible.

Sourced from Docs by LangChain (MCP)

Anything that is objectively right or wrong and cheap to check deterministically becomes a hard assertion. Examples are output that parses against the schema, a required tool call that was made, no forbidden content, and no leaked system prompt. These should never be flaky, so they block on the first failure. Qualities that need judgment, such as helpfulness, faithfulness to context or tone, are scored by an LLM-as-judge or an agentevals trajectory evaluator and logged with t.log_feedback or ls.logFeedback. Logging alone does not fail a test. Soft scores only block when I add an explicit threshold assertion, and I add one only after measuring the judge's variance, so noise does not turn into red builds.

Sourced from Docs by LangChain (MCP)

I start from data. I run the current main branch several times with repetitions and look at how much each example's judge score varies. I set the threshold below that observed band rather than at the mean. For noisy examples I gate on the mean or majority across repetitions instead of a single sample. Where per-example scores are inherently noisy, I gate on an aggregate compared with the baseline experiment, because a small drop across many examples is a real signal. I also validate the judge against a human-labelled sample from an annotation queue. Thresholds are versioned in code, so changing one is a reviewed decision. I revisit them whenever the judge model or the judge prompt changes.

Sourced from Docs by LangChain (MCP)

I separate infrastructure errors from quality failures. Model and judge calls get bounded retries with backoff for retryable errors like rate limits and timeouts. If a call still fails, the test should surface it as an infrastructure error and not as a failed quality assertion, for example through a distinct error type or marker that CI can report separately. I cap concurrency to stay under provider rate limits. The cache removes calls for unchanged cases. For an outage, the team needs an explicit policy: rerun the job, or allow a documented override. The gate should never silently pass, because passing on missing results is how real regressions slip through.

Sourced from Docs by LangChain (MCP)

The integration fits evals into the workflow engineers already use. Tests live next to the code, run with the normal test command, use ordinary assertions and fixtures, and produce a standard exit code that CI and branch protection understand. Each test file is still synced to LangSmith, where the suite becomes a dataset and each run becomes an experiment with outputs, feedback and traces, so nothing is lost for review. I reach for evaluate() when I want dataset-first workflows: large curated datasets, backtesting against production traces, or pairwise experiment comparison driven from a notebook. Many teams use both. evaluate() covers broad benchmarking, and the test integrations cover fast, code-reviewed regression gates.

Sourced from Docs by LangChain (MCP)

I start from the experiment link in the CI log and open the comparison view against the last green main-branch experiment. That shows which examples regressed and which feedback keys dropped. For each failing example, I open the trace to see whether the problem is in retrieval, a tool call, the prompt, or the model's final answer. If the judge's verdict looks wrong, I send the case to an annotation queue so a human can confirm it. A real regression gets fixed, and a newly discovered failure mode becomes a permanent test case. A miscalibrated judge gets new few-shot examples or a rubric fix instead of a code change.

Sourced from Docs by LangChain (MCP)

The suite name determines which dataset the run upserts into. If the name changes per branch or per run, every CI job creates a new, separate dataset. Experiments would then never share examples, and the comparison view against the main-branch baseline would have nothing to line up. A stable name keeps one dataset per logical suite, so each pull request's experiment sits next to main's experiments on the same examples. Use one stable name per suite, for example per service or per agent. Do not encode branch names in it. Branch and commit context belongs in the CI log and the experiment link, not in the dataset identity.

Sourced from Docs by LangChain (MCP)

The same suite runs on every push to main, with the same stable suite name. That produces a stream of baseline experiments on the same dataset. Reviewers compare a pull request's experiment with the most recent main experiment in the comparison view. For the baseline to be fair, the main run must use the same prompt tags, model, judge configuration and dataset version as the pull request, apart from the change under test. If the PR adds or edits test cases, those examples have no baseline, and reviewers judge them on their own merits. I also run main periodically without the cache, so the baseline reflects live model behaviour and not only replayed responses.

Sourced from Docs by LangChain (MCP)

Prompts in the Prompt Hub are versioned with commits and tags. Pulling by tag makes the version under test explicit and reproducible. If CI took the latest commit, someone editing a prompt in the Playground could change what main's baseline measures without any code change, and a PR's regression could actually be someone else's prompt edit. Pulling by tag also makes promotion a reviewable step. A prompt change is tested by pointing the PR at a new commit or a candidate tag, and the eval decides whether that tag should move. The trace records which prompt was used, so reviewers can tie a regression to a specific prompt version.

Sourced from Docs by LangChain (MCP)

I estimate model calls as examples × repetitions × (application calls per example + judge calls per example). For illustration, 200 examples, 2 repetitions, one app call and two judge calls give 200 × 2 × 3 = 1,200 calls per uncached run. Multiply that by PRs and pushes per day. Then I cut it down. LANGSMITH_TEST_CACHE replays unchanged calls instead of re-billing them. A small blocking smoke suite runs on every PR, and the full suite runs on merge to main or nightly. Path filters skip evals when only unrelated code changed. Repetitions are reserved for the examples that actually vary. Judges use cheaper models where agreement with human labels is acceptable.

Sourced from Docs by LangChain (MCP)

On GitHub Actions, workflows triggered by pull requests from forks do not receive repository secrets, so LANGSMITH_API_KEY and the model keys are missing. Exposing them to untrusted code would be a credential leak, because a malicious PR could exfiltrate them. A safe pattern is to require a maintainer to approve or label the PR, after which a trusted workflow runs the evals. Another is to run evals only after the code lands in a branch of the main repository. I avoid patterns that check out and execute fork code with secrets available. The required check should report clearly that evals were not run, rather than showing green.

Sourced from Docs by LangChain (MCP)

Test cases live in code and in a dataset that many engineers can read, so they should be curated or synthetic, not raw customer data. When cases come from production traces, I scrub or anonymise them before they become test parameters. Traces capture full inputs and outputs, so I mask sensitive fields before they leave the process, using the SDK's input and output masking options or by not logging those fields. API keys stay in CI secrets, are scoped to the right workspace and are never printed. Access to the workspace that holds eval datasets follows least privilege. I also check data residency and retention settings against the organisation's policy.

Sourced from Docs by LangChain (MCP)

I treat the judge as a model that needs its own evaluation. I sample cases, have humans score them in an annotation queue against the same rubric, and measure how often the judge agrees with them. Where it disagrees, I tighten the rubric, add few-shot examples to the judge prompt, or switch to a stronger judge model. I pin the judge model and the judge prompt version, so the measurement does not drift. I check for known biases, such as favouring longer answers or position bias in pairwise setups. Until agreement is good, the judge score stays non-blocking and visible only, and hard assertions do the gating.

Sourced from Docs by LangChain (MCP)

Per-example assertions are simple and map naturally to test runners: one bad case gives one red test with a clear trace. But they are sensitive to noise on borderline examples. They also miss broad, small degradations where each example stays just above its threshold. Aggregate comparison against the main baseline catches that drift, but it needs a post-run step that reads experiment results and compares them, and the failure is less localised. I use both. Critical cases get per-example hard assertions. Soft quality gets an aggregate check with a tolerance based on measured variance. The comparison view then shows exactly which examples drove the aggregate drop.

Sourced from Docs by LangChain (MCP)

Disputed or ambiguous results go to an annotation queue where reviewers score runs against a rubric. When reviewers confirm a new failure mode, it is exported to a dataset or, more often in this workflow, added as a new parametrized test case with inputs and reference outputs in code. That way it runs on every future PR. Production issues follow the same path: a failing trace is captured, scrubbed of PII and turned into a regression test. The rule is that every confirmed bug leaves behind a test. Over time the suite reflects real failure modes rather than only the cases engineers imagined up front.

Sourced from Docs by LangChain (MCP)

Changing the judge changes the measuring instrument, so old and new scores are not directly comparable. I run the new judge alongside the old one on the same experiments, logging both as separate feedback keys, and compare each against human labels from an annotation queue. I recalibrate thresholds from the new judge's score distribution on main. Then I switch the blocking assertion to the new key in a dedicated PR, and re-run main to create a fresh baseline. I never bundle a judge upgrade with an application change, because a score movement could then come from either one.

Sourced from Docs by LangChain (MCP)

A single gate has to be both strict and stable, and those goals conflict for subjective metrics. The blocking suite holds deterministic assertions and well-calibrated judge checks with proven agreement with humans. It runs fast on every PR and is a required status check. The non-blocking suite holds new tests, experimental judges, quarantined flaky cases and expensive long-horizon agent scenarios. It still syncs to LangSmith as experiments, so reviewers see trends, but it does not stop merges. Tests graduate from non-blocking to blocking once they have shown stable results across repeated runs on main.

Sourced from Docs by LangChain (MCP)

They answer different questions. CI evals are offline: a fixed dataset, run before merge, to catch regressions caused by code or prompt changes. Online evaluators run on production runs or threads, with filters and sampling rates, to catch issues from real traffic that the dataset never covered. They feed dashboards, alerts, automation rules and webhooks. They form a loop. Online evaluation surfaces new failure modes, those become test cases, and CI stops them from coming back. One operational difference to know: running an online evaluator on a trace upgrades that trace to extended retention, which has cost and data-retention implications.

Sourced from Docs by LangChain (MCP)

It should fail closed by default, because a gate that passes when it could not evaluate lets regressions merge. But it must fail clearly as an infrastructure problem, not as a quality regression, so nobody wastes time debugging prompts. Retries with backoff handle short blips. For long outages, the team needs a documented break-glass path: an authorised person can override the required check with a recorded reason, and the suite is rerun once the provider recovers. Cached calls still replay, so only uncached cases are affected. I would also alert when the infrastructure error rate spikes, separately from eval failures.

Sourced from Docs by LangChain (MCP)

The failing tests are doing their job. The fix is to update the test expectations, not to loosen the gate. In the same PR, I update reference outputs, assertions or thresholds for the affected cases, so the reviewer sees the behaviour change and the expectation change together. Reviewers confirm in the comparison view that only the intended examples moved. Unexpected movers are investigated separately. After merge, the next main run produces the new baseline experiment. I avoid broad threshold drops to get green, because they also lower protection on unrelated examples.

Sourced from Docs by LangChain (MCP)

For agents, a correct final answer can hide a bad path: redundant tool calls, a tool called with the wrong arguments, or a lucky guess with no retrieval at all. The opposite also happens, where the path was correct but the final wording drifted. Trajectory evaluators in agentevals score the sequence of steps and tool calls, either against a reference trajectory or with a trajectory LLM-as-judge. I use deterministic trajectory checks, such as which required tool calls happened, as hard assertions. I use the judge-based trajectory score as feedback. This is especially useful for LangGraph agents, where the trace already contains every node and tool call.

Sourced from Docs by LangChain (MCP)

First I confirm that it is flaky by running it with repetitions on an unchanged main. If it is, I move it from the blocking suite to a non-blocking suite that still runs and syncs to LangSmith as an experiment, so its scores remain visible and comparable over time. I open a ticket with an owner and a deadline. Then I fix the root cause: a vague rubric, a judge that disagrees with humans, a non-deterministic tool, or a threshold too close to the mean. Deleting the test or wrapping it in retries until it passes hides the problem. Quarantine is only acceptable if it is tracked and temporary.

Sourced from Docs by LangChain (MCP)

Online evaluators and dashboards surface problem runs. I add those runs to an annotation queue, where a reviewer confirms the failure and writes the correct reference output. Before a case enters the test suite, I remove or replace personal and confidential data, because tests and datasets are widely visible and live in version control. I keep only the inputs needed to reproduce the behaviour. If the case depends on retrieved documents or tool results, I either freeze those as fixtures or confirm that the live dependency is stable. Each new case should carry a short comment explaining which incident or failure mode it guards against.

Sourced from Docs by LangChain (MCP)

With this integration, the test parameters in code are the source of truth, and each run upserts the dataset from them. Manual UI edits to that dataset can diverge from what the code produces, which makes it unclear which reference output a given experiment used. The edits may also be overwritten or ignored on the next sync. To avoid this, I treat the synced dataset as derived and read-only. Changes to inputs or reference outputs go through a pull request on the test file, so they are reviewed and versioned with the code. Curated datasets that people edit in the UI live separately and feed evaluate() runs, not the CI suite.

Sourced from Docs by LangChain (MCP)

Each CI run creates its own experiment against the shared dataset, so concurrent PRs do not overwrite each other's results. Reviewers follow the experiment link from their own job. The risks are elsewhere. Concurrent jobs share provider rate limits and can trigger 429s, so I cap concurrency per job and retry with backoff. If two PRs change test parameters for the same suite, the dataset reflects whichever synced last, so baselines on changed examples need care. Shared cache directories should be scoped so that one branch's replayed responses do not mask another branch's real changes. Overall spend also scales with concurrent PRs.

Sourced from Docs by LangChain (MCP)

Absolute judge scores are weak when two outputs are both acceptable. I use pairwise comparison instead. A pairwise annotation queue takes exactly two experiments, the PR and the main baseline, and reviewers choose the better output per example against a rubric. A pairwise LLM-as-judge can give a quicker first pass, ideally with position swapping to reduce bias. If humans prefer the new behaviour, I update the reference outputs or thresholds in the test code within the same PR, so the new baseline is explicit and reviewed. If preferences are split, I look for the specific example categories that regressed before approving.

Sourced from Docs by LangChain (MCP)

The cache is valuable on PRs because it replays unchanged LLM calls instead of re-billing them and makes reruns faster and more stable. But replayed responses do not reflect what the live model would do today. I bypass it in a scheduled job, for example nightly or before releases, so the suite exercises live model behaviour and catches provider-side changes. I also bypass or clear it when changing models, model parameters or anything that affects requests, and when debugging a suspected flake. The live runs are also the honest baseline, so I compare cached PR results with them periodically.

Sourced from Docs by LangChain (MCP)

A bare number gives reviewers nothing to argue with. I have the judge return a short explanation along with its score, and I log it as the feedback comment next to the score. I trace the judge call, so reviewers can see the exact judge prompt, model and input it saw. I pin the judge prompt in the Prompt Hub by tag, which ties every score to a specific judge version. In the comparison view, a reviewer can then open a regressed example, read the app trace and the judge's reasoning, and decide whether the regression is real or a judge mistake. Disputed cases go to the annotation queue.

Sourced from Docs by LangChain (MCP)

Each test case's inputs hold the conversation history up to the turn under test, plus the new user message. The reference output describes the expected response or behaviour. The test replays the history through the real chain or LangGraph agent and asserts on the final turn. For behaviour that only appears across several turns, I script a short simulated conversation within one test and log feedback per turn or for the whole exchange. I keep user simulators deterministic, or at least repeated, because a simulated user adds a second source of non-determinism. In production, threads group multi-turn conversations, so online evaluators can score whole threads too.

Sourced from Docs by LangChain (MCP)

I give each feature its own suite with a stable LANGSMITH_TEST_SUITE name and a clear owning team. Datasets and experiments then stay focused and comparable. CI uses path filters, so a PR runs only the suites whose code, prompts or dependencies changed, plus a small shared smoke suite. Cache directories are scoped per suite. Concurrency limits are set per job to respect shared provider quotas, and spend is tracked per suite. Shared judges and rubrics live in one internal library with pinned versions. Ownership rules make clear who triages a red suite, so failures do not sit untouched in someone else's queue.

Sourced from Docs by LangChain (MCP)

Code rules are deterministic, free to run, fast and easy to debug: schema validation, regex for forbidden content, checking that a tool was called. They make ideal blocking assertions, but they cannot judge meaning, tone or faithfulness. LLM-as-judge covers those qualities with prebuilt templates and few-shot examples. But it costs model calls, adds latency and noise, can carry biases, and needs its own validation against human labels. In practice, I push as much as possible into code rules and use judges for the remainder. Judge scores gate only after calibration, preferably on aggregates or repeated samples rather than single draws.

Sourced from Docs by LangChain (MCP)

1. ASetting LANGSMITH_TRACING=false
2. BNaming the file eval_*.py
3. CDecorating it with @pytest.mark.langsmithCorrect answer
4. DCalling evaluate() inside the test

Why: The @pytest.mark.langsmith marker tells the plugin to log the test case as a dataset example and record the run as an experiment. Naming the file or calling evaluate() inside a test does not do this.

Sourced from Docs by LangChain (MCP)

1. A@langsmith.experiment
2. B@pytest.mark.langsmithCorrect answer
3. C@traceable(suite=True)
4. D@pytest.mark.evaluate

Why: The @pytest.mark.langsmith marker tells the integration to sync that test as an example and record its result in an experiment.

Sourced from Docs by LangChain (MCP)

1. Alangsmith/traceable
2. B@langchain/vitest-evals
3. Clangsmith/evaluation
4. Dlangsmith/vitestCorrect answer

Why: The Vitest integration is imported from langsmith/vitest and used together with the LangSmith reporter.

Sourced from Docs by LangChain (MCP)

1. AIt fails only if the judge errored
2. BIt passes; the feedback is recorded but does not fail the testCorrect answer
3. CIt is marked flaky and retried
4. DIt fails because the score is below 0.5

Why: Logged feedback is only recorded; the test fails only when an assertion, such as assert score >= threshold, fails.

Sourced from Docs by LangChain (MCP)

1. AAn online evaluator posts a webhook that closes the PR
2. BThe experiment is marked failed and reviewers must delete it
3. CLangSmith calls the Git provider's API to block the PR
4. DA failed assertion makes the runner exit non-zero, the CI check turns red, and branch protection blocks the mergeCorrect answer

Why: Blocking comes from the CI check status enforced by branch protection, and the check status is driven by the runner's exit code.

Sourced from Docs by LangChain (MCP)

1. AWhether the output parses against the required JSON schemaCorrect answer
2. BWhether the summary is the most concise possible
3. CWhether the tone matches the brand
4. DWhether the answer sounds empathetic

Why: Schema validity is deterministic and objectively checkable, so it can block without adding noise.

Sourced from Docs by LangChain (MCP)

1. AAs a quality regression in the model's output
2. BAs an infrastructure error, reported distinctly, without silently passingCorrect answer
3. CBy dropping the example from the dataset
4. DAs a passed test

Why: Separating infrastructure errors from quality failures avoids both false regressions and false passes.

Sourced from Docs by LangChain (MCP)

1. AAn annotation queue reviewer rejects the run
2. BLangSmith sends a webhook that cancels the job
3. CThe experiment is marked red in the UI only
4. DA failed assertion makes the test runner exit non-zero, which CI treats as failureCorrect answer

Why: Gating relies on the normal test runner exit code from failed assertions. The LangSmith sync records results for review but does not itself control the build.

Sourced from Docs by LangChain (MCP)

1. ALANGSMITH_DATASET_ID
2. BLANGSMITH_TEST_SUITECorrect answer
3. CLANGSMITH_EXPERIMENT
4. DLANGSMITH_PROJECT

Why: LANGSMITH_TEST_SUITE sets the suite name, which becomes the dataset that each run's experiment is recorded against.

Sourced from Docs by LangChain (MCP)

1. ACaches the dataset so it is not upserted
2. BKeeps traces locally instead of sending them to LangSmith
3. CStores judge thresholds between runs
4. DReplays unchanged LLM calls instead of re-billing themCorrect answer

Why: The cache records request and response pairs, so identical calls are replayed rather than sent to the provider again.

Sourced from Docs by LangChain (MCP)

1. AA dataset
2. BAn exampleCorrect answer
3. CA thread
4. DA project

Why: Each test case becomes one example, with inputs and reference outputs taken from the test parameters.

Sourced from Docs by LangChain (MCP)

1. AA new experiment on the suite's datasetCorrect answer
2. BA new prompt commit
3. CA new annotation queue
4. DA new dataset

Why: The dataset is upserted and stays stable, while each run is recorded as a new experiment containing outputs, feedback and traces.

Sourced from Docs by LangChain (MCP)

1. ABecause untagged prompts cannot be traced
2. BTags automatically raise judge scores
3. CTags are faster to download than commits
4. DSo the prompt version under test is explicit and reproducibleCorrect answer

Why: Pulling by tag pins the exact prompt version, so a score change can be tied to a known prompt version rather than an unreviewed edit.

Sourced from Docs by LangChain (MCP)

1. ARun untrusted fork code in a privileged workflow that has secrets
2. BRequire maintainer approval or a trusted workflow before evals run, and report clearly when evals were skippedCorrect answer
3. CMark the eval check as passed for forks
4. DHardcode LANGSMITH_API_KEY in the workflow file

Why: This protects credentials from untrusted code and keeps the gate from showing a false green.

Sourced from Docs by LangChain (MCP)

1. ARaise the threshold so it fails more often
2. BSwitch to a random judge model each run
3. CDelete the example
4. DGate on the mean across repetitions with a threshold set from measured varianceCorrect answer

Why: Averaging repetitions and calibrating against observed variance separates noise from real drops.

Sourced from Docs by LangChain (MCP)

1. A900Correct answer
2. B300
3. C500
4. D600

Why: 100 examples × 3 repetitions × (1 + 2) calls = 900 model calls.

Sourced from Docs by LangChain (MCP)

1. AThe main-branch baseline experimentCorrect answer
2. BA random production project
3. CThe annotation queue
4. DThe first experiment ever run on the dataset

Why: Comparing with main's latest experiment isolates the effect of the PR's changes.

Sourced from Docs by LangChain (MCP)

1. ARe-run until the build passes
2. BDelete the failing tests
3. CSend the cases to an annotation queue for human review against a rubricCorrect answer
4. DLower all thresholds to zero

Why: Annotation queues resolve disputed cases with human judgment, and the results can calibrate the judge.

Sourced from Docs by LangChain (MCP)

1. AExported to the dataset or added as a new test case so future PRs check itCorrect answer
2. BAs a new LangSmith project
3. CBy raising the judge's temperature
4. DAs a comment on the pull request only

Why: Adding confirmed failures to the dataset or the tests makes the suite grow and prevents the regression from returning.

Sourced from Docs by LangChain (MCP)

1. AA stricter schema assertion
2. BAn aggregate score comparison against the baseline experimentCorrect answer
3. CA larger cache directory
4. DRemoving repetitions

Why: Broad small degradation shows up in aggregates compared with the baseline, even when no single example crosses its threshold.

Sourced from Docs by LangChain (MCP)

1. AExperiments stop appearing in the UI
2. BCached LLM responses can hide regressions after prompts or models changeCorrect answer
3. CDatasets get deleted
4. DTraces are moved to extended retention

Why: The cache replays earlier LLM responses. If it is not invalidated when prompts, models or dependencies change, tests can pass against outputs the new code would never produce.

Sourced from Docs by LangChain (MCP)

1. At.log_reference_outputsCorrect answer
2. Bt.log_expected
3. Ct.log_feedback
4. Dt.log_outputs

Why: t.log_reference_outputs stores the reference outputs that become part of the dataset example.

Sourced from Docs by LangChain (MCP)

1. Als.recordMetric
2. Bls.logFeedbackCorrect answer
3. Cls.expectScore
4. Dls.feedback.create

Why: ls.logFeedback is the Vitest equivalent of t.log_feedback in pytest.

Sourced from Docs by LangChain (MCP)

1. AAny number of experiments
2. BExactly two experimentsCorrect answer
3. CExactly one experiment and one dataset
4. DOnly production projects

Why: Pairwise queues compare exactly two experiments, for example a PR experiment against the main baseline.

Sourced from Docs by LangChain (MCP)

1. AagentevalsCorrect answer
2. Bopenevals-trajectory
3. Clanggraph-test
4. Dpytest-langsmith-agents

Why: agentevals provides trajectory evaluators that score an agent's sequence of steps and tool calls.

Sourced from Docs by LangChain (MCP)

1. AWraps the OpenAI client so its calls are traced under the test caseCorrect answer
2. BConverts OpenAI outputs into reference outputs
3. CRetries failed OpenAI calls forever
4. DMocks OpenAI so tests run offline

Why: wrap_openai instruments the client, so every model call appears in the trace recorded for that test.

Sourced from Docs by LangChain (MCP)

1. AAdding it to a CI test suite
2. BRunning an online evaluator on itCorrect answer
3. CTagging its prompt in the Prompt Hub
4. DViewing it in the comparison view

Why: The reference notes state that running an online evaluator on a trace upgrades it to extended retention.

Sourced from Docs by LangChain (MCP)

1. ADataset splits
2. Bpytest test suites
3. COnline evaluators on production runs or threadsCorrect answer
4. DPrompt Hub tags

Why: Online evaluation runs on production traffic and uses filters and sampling rates to choose which runs to score.

Sourced from Docs by LangChain (MCP)

1. ATo avoid creating feedback
2. BCached runs cannot create experiments
3. CTo measure live model behaviour and catch provider-side changes that replayed responses would hideCorrect answer
4. DBecause the cache deletes the dataset

Why: Replayed responses reflect past behaviour; live runs confirm that the baseline still matches what the model does now.

Sourced from Docs by LangChain (MCP)

1. AStore datasets in a public repository
2. BUse raw production transcripts as test parameters
3. CUse curated or scrubbed inputs and mask sensitive fields before they are sentCorrect answer
4. DPrint API keys in logs for debugging

Why: Tests and traces are widely visible, so data should be sanitised and masked at the source.

Sourced from Docs by LangChain (MCP)

1. AThe second run overwrites the first experiment
2. BEach run records its own experiment on the shared datasetCorrect answer
3. CLangSmith rejects the second run
4. DBoth runs merge into one experiment

Why: Experiments are created per run, so concurrent runs stay separate and each can be compared with the baseline.

Sourced from Docs by LangChain (MCP)

1. AMulti-turn conversationsCorrect answer
2. BPrompt commits
3. CCI jobs
4. DExperiments on the same dataset

Why: Threads group the runs of a multi-turn conversation, and online evaluators can score whole threads.

Sourced from Docs by LangChain (MCP)

1. AExport traces from annotation queues
2. BFan out via an OpenTelemetry collectorCorrect answer
3. CDuplicate @traceable decorators
4. DUse two LANGSMITH_TEST_SUITE values

Why: LangSmith accepts OTLP, and a collector can fan traces out to several destinations.

Sourced from Docs by LangChain (MCP)