Evaluating agent trajectories

Because the final answer can be right for the wrong reasons. A support agent might issue a refund without verifying the order, or call the CRM five times in a loop, and still write a polite, correct-looking reply. Those are safety and cost problems that answer-only checks hide. So I score two things separately. First, the tool path, with an agentevals trajectory match against a reference sequence, or a trajectory LLM-as-judge where several paths are valid. Second, the final reply, with code rules and a correctness judge. Keeping them as separate feedback keys in LangSmith means a regression in the path stays visible even when answer scores look flat. That is usually the earliest signal that a prompt or model change has broken the agent's reasoning.

Sourced from Docs by LangChain (MCP)

The production support agent, built on LangGraph, is traced into a LangSmith project with LANGSMITH_TRACING enabled, and threads group multi-turn tickets. Representative and failed tickets go to a single-run annotation queue. There, reviewers record the reference tool-call sequence and the expected reply, then export the examples to a dataset. Offline, evaluate() runs each agent version against that dataset with mocked tools and repetitions. Two families of evaluators score the run: trajectory evaluators from agentevals, using match or LLM-as-judge, and final-answer evaluators, using code rules and an LLM judge. Experiments are compared with a baseline, and CI runs the same cases through pytest. Online evaluators sample production runs, and automation rules push low-scoring traces back into the annotation queue, which closes the loop.

Sourced from Docs by LangChain (MCP)

I treat safety-critical ordering as a hard gate, not a soft score. Refund tickets use strict trajectory matching, or a dedicated code evaluator that checks the issue-refund call comes after a successful order verification with matching arguments. I never use unordered mode for these examples, because unordered mode would accept the wrong order. The final-answer score cannot offset a trajectory failure. CI blocks the merge when that critical-path score regresses, even if the reply looks perfect. When a ticket has several valid paths, the LLM judge may score flexibility, but the ordering rule still runs alongside it as a deterministic check.

Sourced from Docs by LangChain (MCP)

Mocking keeps refunds and CRM writes deterministic and safe. An evaluation run must never move real money or change customer records, and fixed tool responses make experiments comparable across agent versions. The risk is drift between the mocks and the real systems. If the real refund API adds a field or a new error, the mocks hide it and the agent looks better than it is. I limit that risk in three ways. I build mock responses from real traced tool outputs. I include error and edge responses such as order not found or partial refund. I rely on online evaluation of production runs to catch behaviour the mocks do not cover.

Sourced from Docs by LangChain (MCP)

I use deterministic matching when there is one correct path or a small set of acceptable shapes. Strict, unordered, subset or superset modes express those cheaply and reproducibly. I use create_trajectory_llm_as_judge when several paths are genuinely valid. For example, the agent might look up the customer first or the order first, or skip a step when context already contains the data. I pass the reference trajectory so the judge reasons against a known-good path, and I add few-shot examples from human-labelled cases to anchor its judgement. The judge costs more and is less deterministic, so I use it selectively. I still keep hard code checks for safety ordering, because a judge should never be the only guard on refunds.

Sourced from Docs by LangChain (MCP)

The test cases live in pytest files marked with @pytest.mark.langsmith, so each pull request run creates a LangSmith experiment linked to the dataset. Tools are mocked, so CI is safe. I separate gates by severity. Critical-path checks block the merge if they regress: verify before refund, correct refund arguments, no unsafe writes, no leaked IDs. Softer metrics, such as LLM-judge answer quality, are reported on the pull request and compared with the baseline, but they only warn unless the drop is large and confirmed. That keeps CI strict where safety matters without blocking every change because of judge noise.

Sourced from Docs by LangChain (MCP)

Support tickets contain names, emails, addresses and order details, so I minimise exposure at every stage. Before traces leave the application, I mask or redact sensitive fields using the tracing SDK's masking options, keeping only what evaluation needs. Datasets exported from annotation queues get a second scrub, with real identifiers replaced by synthetic but consistent ones so that the mocked tools still line up. Access to the project, the queues and the datasets is limited to the people who need it. I align retention with policy, remembering that online evaluators upgrade traces to extended retention, so sampling also affects how long data is kept. Code evaluators check that replies do not leak internal IDs.

Sourced from Docs by LangChain (MCP)

I pick the mode per flow based on what is actually wrong for that ticket type. For safety-critical sequences, like verify identity, then look up the order, then refund, I use strict, because order matters. Where the order is flexible but every call is required, I use unordered. Superset suits cases where the agent must make the reference calls but extra read-only lookups are fine. Subset suits cases where the agent must stay within an allowed set of tools and not go beyond it. I also decide how tool arguments are compared, because a matching tool name with the wrong order ID is still a failure. When no fixed sequence captures 'good', I use the trajectory LLM-as-judge instead, calibrated against human labels.

Sourced from Docs by LangChain (MCP)

I start from production traces rather than imagined cases. I filter the support project for representative tickets and for failures, then send them to an annotation queue. There, reviewers record the correct tool sequence, its key arguments and the expected reply, and export the result to a LangSmith dataset. I tag examples by category and criticality so I can slice the results. Online evaluators keep feeding the dataset: low-scoring sampled runs go back to the queue through automation rules. For maintenance, when a tool is renamed or split, I update the references in one pass and rerun the baseline experiment. That way the false failures don't pile up and erode trust in the gate.

Sourced from Docs by LangChain (MCP)

Tool names alone are not enough. A refund call with the wrong order ID or amount is a wrong path. In create_trajectory_match_evaluator I set how arguments are compared. Safety-relevant arguments such as order ID and refund amount use exact matching. Volatile fields such as timestamps, request IDs or free-text notes are ignored or compared with a custom rule. Per-tool overrides let me be strict on issue_refund and lenient on search tools, where the query phrasing varies. I document these rules next to the dataset. Loose matching makes the evaluator pass wrong paths, and overly strict matching creates noisy failures that reviewers learn to ignore.

Sourced from Docs by LangChain (MCP)

In evaluate() I set repetitions so each example runs several times against the same agent version. One run can hide a path that appears only occasionally, for example refunding without verification one time in five. I look at results per example, not only the mean. For safety-critical examples the bar is that every repetition passes, because one unsafe path in production is unacceptable. For softer quality metrics, an average or pass rate across repetitions is fine. Examples whose scores vary across repetitions get flagged for inspection, since that variance usually points to an ambiguous prompt or tool description.

Sourced from Docs by LangChain (MCP)

I start from human-labelled cases in the annotation queue, where reviewers marked trajectories as acceptable or not with reasons. Some of these become few-shot examples for create_trajectory_llm_as_judge, and a separate held-out set measures agreement between the judge and the humans. I specifically include tricky negatives, such as plausible replies reached through an unsafe path. If the judge disagrees with humans on those, I revise the rubric or the examples before relying on it. I version the judge prompt so score changes can be attributed to the agent and not to the evaluator. I recheck agreement periodically as new failure types appear.

Sourced from Docs by LangChain (MCP)

They fail independently. An agent can take the correct path and then write a reply that is wrong, leaks an internal ID or promises the wrong amount. An agent can also write a perfect reply after an unsafe path. Separate scores show which part regressed and stop one from masking the other. In this workflow the trajectory score is a gate in its own right: a good answer never compensates for a wrong or unsafe path. Keeping them separate also makes experiment comparison useful. I can see a version that improved reply quality but made the tool path worse, which is the situation step 7 asks us to investigate.

Sourced from Docs by LangChain (MCP)

In LangSmith I open the comparison view with the baseline experiment and the candidate, both run on the same dataset version. I check aggregate scores per evaluator, but the important work is per example. I filter for examples where the trajectory score dropped, even if the answer score held, because that is where unsafe or wasteful paths hide. I also look at examples that improved, to confirm the gain is real and not noise. I then check whether the variance across repetitions changed. Each regression is classified as a real agent bug, an evaluator issue or a reference that needs updating. Regressions are not dismissed as noise without inspection.

Sourced from Docs by LangChain (MCP)

First, I keep blocking gates on deterministic evaluators wherever possible: trajectory match and code rules over mocked tools. Second, I use repetitions and define pass criteria explicitly. A safety check must pass on every repetition, while quality metrics use a threshold on the aggregate. Third, I pin agent and judge prompt versions through Prompt Hub commits or tags, and pin the dataset version, so a run changes only what the pull request changed. Caching in evaluate() avoids paying again for identical calls during reruns. When a test is flaky, I investigate it as a signal of ambiguous agent behaviour rather than simply retrying until it passes.

Sourced from Docs by LangChain (MCP)

I attach online evaluators to the production project and target either runs or threads, depending on whether the check is per step or per conversation. Filters focus evaluation where risk is highest, for example traces that called the refund tool or tickets the customer reopened. Sampling rates control volume and cost. I sample risky segments at a high rate and routine ones at a lower rate. Production has no reference trajectory, so online evaluators are reference-free: rule checks on the tool sequence, such as a refund without a prior verification, and LLM-judge checks on reply quality. Dashboards and alerts track the scores, and automation rules route low-scoring traces to the annotation queue.

Sourced from Docs by LangChain (MCP)

Inputs hold the ticket: the conversation messages, customer context and any metadata the agent receives. Reference outputs hold the expected reply and the reference trajectory, meaning the ordered tool calls with names and arguments. I also store the mock tool responses for that example, so the run is deterministic. Metadata records the example type, such as refund, address change or FAQ, plus the trajectory match mode to use, the risk level and the source: curated, production trace or synthetic. Splits or tags let CI run a critical subset quickly while the full offline run covers everything. Each example therefore says both what to do and how strictly to judge it.

Sourced from Docs by LangChain (MCP)

I layer two evaluators. The trajectory LLM judge, given the reference trajectory and few-shot labels, scores whether the overall path is reasonable, which tolerates harmless variation in order. Alongside it, a deterministic code evaluator encodes the invariants that must hold on every path: a successful verification must come before any refund, the refund arguments must match the verified order, and no CRM write may happen without an identified customer. The example fails if the invariant check fails, regardless of the judge score. This gives flexibility where it is safe and strictness where it is not, which is the core requirement of this workflow.

Sourced from Docs by LangChain (MCP)

I treat it as a likely hidden problem. A stable answer with a worse path can mean the agent skipped verification, made redundant or extra tool calls, or used the wrong tool and got lucky with the mocks. I open each affected example in the comparison view and read the trace step by step. If the new path is unsafe, it is a blocking regression even though the reply looks correct. If the new path is valid but different, the reference or match mode is too strict, so I update it, possibly moving that example to the LLM judge. Extra calls that are safe but wasteful matter for latency and cost, so I log them.

Sourced from Docs by LangChain (MCP)

Strict matching is reproducible but brittle, and it can reject valid paths. LLM judges are flexible but cost more, vary between runs and need calibration. Mocks make runs safe and deterministic but can drift from real systems. References need upkeep whenever policy or tools change. Online evaluation adds cost and extends retention, so sampling is a privacy decision as well as a budget decision. The worst failure mode is false confidence: an evaluator configured too loosely, such as unordered mode on refund tickets, quietly passes unsafe agents. I mitigate this with deterministic invariant checks, per-example match modes, calibrated judges, and the rule that a good answer never offsets a bad path.

Sourced from Docs by LangChain (MCP)

Hard, checkable rules belong in code. Code evaluators are deterministic, cheap and cannot be argued with. Examples: the refund amount stated in the reply equals the order total from the mocked order data. The reply contains no internal IDs, matched against known ID patterns. The reply does not promise actions the trajectory never performed, such as saying a refund was issued when no refund call happened. Policy phrases such as the refund timeline must match the approved wording. The LLM judge is reserved for fuzzy qualities like correctness against the reference output, tone and completeness. That split keeps the judge from being the only guard on rules with exact answers.

Sourced from Docs by LangChain (MCP)

An automation rule watches online evaluator scores. When a trace scores low, for example a flagged tool ordering or a poor reply, the rule pushes that trace into the single-run annotation queue. Webhooks can also notify the owning team. Reviewers confirm whether it is a real failure, write the correct reference tool-call sequence and expected reply, and export it to the dataset. The next offline experiment and every CI run then include that case, so the production failure becomes a permanent regression test. The dataset grows from real failures rather than only from imagined scenarios.

Sourced from Docs by LangChain (MCP)

Threads group the runs of one conversation, so I can see the whole ticket: the customer asks, the agent verifies, the customer gives an order number, the agent refunds. For offline examples I store the conversation history up to the decision point as input, and the reference trajectory and reply for the next turn as the expected output. Some examples also cover the full ticket. Online, I attach thread-level evaluators for outcomes that only make sense across turns, such as whether the issue was resolved or whether the agent asked for verification before acting. Run-level evaluators still catch step-level problems such as a wrong tool argument.

Sourced from Docs by LangChain (MCP)

I build it from first principles instead of quoting numbers. Agent runs equal the number of examples multiplied by the number of repetitions. Each agent run costs its model calls, which I can read from existing traces as average tokens per ticket. Mocked tools add little. Evaluator cost comes on top: trajectory match and code evaluators are cheap, while every LLM-as-judge call adds roughly one more model call per evaluator per run. Duration is roughly total runs multiplied by average latency, divided by concurrency, subject to provider rate limits. Caching cuts the cost of reruns. This tells me whether to apply the LLM judge only to multi-path examples, and how much concurrency CI can afford.

Sourced from Docs by LangChain (MCP)

Every experiment should record exactly what was tested. I pin the dataset version, because LangSmith datasets are versioned as examples change. I pin prompts by Prompt Hub commit or tag, for both the agent and the LLM judges, so a judge prompt edit does not appear as an agent change. I record the agent code commit and model settings in experiment metadata, and the mocks are versioned alongside the dataset. When comparing against a baseline, I rerun the baseline on the current dataset version if references changed. Otherwise the comparison mixes dataset drift with agent change.

Sourced from Docs by LangChain (MCP)

That is a signal, not something to hide. If the agent calls lookup_order with an ID that is not in the mock data, the mock should return a realistic error such as order not found, not invent a success. The trajectory evaluator will usually flag the unexpected argument as well. For unscripted write tools, such as a refund on the wrong order, the mock records the call and returns a plausible response, so the trajectory evaluator can fail the example clearly. I never let mocks fall through to real systems. Recurring unscripted calls show either an agent bug or a gap in the example's mock data, and reviewers resolve which one it is.

Sourced from Docs by LangChain (MCP)

I weigh coverage against cost and retention. Every evaluated trace costs evaluator compute, LLM-judge calls are the expensive part, and running an online evaluator upgrades the trace to extended retention, which has cost and privacy implications. I use filters to target risk. Traces that touched refund or CRM write tools get a high sampling rate, while low-risk FAQ traffic gets a small sample to watch for drift. I start conservatively and raise sampling for a new agent version during rollout. Afterwards I lower it once scores are stable. Cheap rule-based checks can run at higher rates than LLM judges.

Sourced from Docs by LangChain (MCP)

Datasets drift as policies, tools and customer behaviour change. I schedule reviews where the owners of refund and CRM policy confirm that the reference trajectories still reflect current rules. When a tool is renamed or a verification step is added, I update the references in one deliberate change and rerun the baseline. To avoid overfitting, I keep a held-out portion that prompt authors do not tune against. I also keep adding fresh production cases through the annotation queue. I track coverage by ticket type, so the dataset does not end up dominated by whatever failed most recently.

Sourced from Docs by LangChain (MCP)

I build a dataset from a window of production traces, using their inputs, with references taken from human-reviewed outcomes where those exist. The candidate agent runs against it with evaluate() and mocked tools. I then compare its trajectories and replies with what the production agent did on the same inputs. Reference-free checks, such as ordering invariants and leaked IDs, run on every example. Reference-based checks run where reviewers supplied a reference. Backtesting shows how the candidate would have handled real traffic before rollout. The mocks must reproduce the tool responses recorded in those traces, otherwise the comparison is not fair.

Sourced from Docs by LangChain (MCP)

From online evaluators, I track the rate of unsafe tool orderings, such as a refund without prior verification, as a top-level metric with an alert. I also track the reply-quality judge scores, the rate of leaked internal IDs, and tool error rates. Operational metrics such as latency, tokens per ticket and the number of tool calls per ticket help catch wasteful paths. I segment by ticket type and agent version, so a regression after a rollout is visible. Alerts on safety metrics page the owners. Quality drops feed automation rules that queue traces for review instead of waking anyone up.

Sourced from Docs by LangChain (MCP)

Pairwise queues compare exactly two experiments side by side. I use them when automatic scores are close or disagree, for example when a new prompt changes both the path and the wording. Reviewers see the same ticket handled by the baseline and by the candidate, and choose which handled it better, and why. This is useful for subjective reply quality and for ambiguous multi-path cases where the LLM judge is uncertain. The decisions also become calibration data for the judges. They do not replace hard safety checks; they are for deciding between two acceptable candidates.

Sourced from Docs by LangChain (MCP)

I configure the single-run queue with a reviewer count, so important cases such as refund disputes get more than one reviewer. Reservations stop two people from working on the same item at the same time. Rubrics and assertions make the labels consistent: reviewers record the reference tool sequence in a fixed format and answer explicit questions such as whether verification came before the refund. When reviewers disagree, the item goes to a policy owner, and the resolution feeds back into the rubric. I prioritise the queue by risk, so safety-related traces are labelled before routine quality samples.

Sourced from Docs by LangChain (MCP)

LangGraph is traced natively. For other Python code, such as a custom order-verification service, I add the @traceable decorator to the relevant functions, and wrappers like wrap_openai capture direct model calls. Integrations cover frameworks such as the Vercel AI SDK. If services already emit OpenTelemetry, LangSmith accepts OTLP directly. An OpenTelemetry collector can fan the same spans out to LangSmith and to the existing observability backend. The goal is that every tool call appears in the trace with its arguments, because trajectory evaluation is only as good as the tool calls we capture.

Sourced from Docs by LangChain (MCP)

1. ASubset
2. BSuperset
3. CStrictCorrect answer
4. DUnordered

Why: Strict mode requires an exact sequence match, including order.

Sourced from Docs by LangChain (MCP)

1. AUnorderedCorrect answer
2. BStrict
3. CA code invariant check for verify-before-refund
4. DNone of the modes; all of them fail it

Why: Unordered mode only checks that the same calls were made, so it ignores the unsafe order.

Sourced from Docs by LangChain (MCP)

1. ASo refunds and CRM writes are deterministic and safeCorrect answer
2. BTo make the agent's replies shorter
3. CTo avoid the need for reference outputs
4. DTo turn on extended retention

Why: The workflow mocks tools so evaluation never performs real refunds or writes, and so runs are repeatable.

Sourced from Docs by LangChain (MCP)

1. A@pytest.mark.langsmithCorrect answer
2. B@pytest.mark.trace
3. C@pytest.mark.parametrize_dataset
4. D@pytest.mark.agentevals

Why: The reference notes name @pytest.mark.langsmith as the pytest integration.

Sourced from Docs by LangChain (MCP)

1. AInspect each example where the trajectory score droppedCorrect answer
2. BDelete those examples from the dataset
3. CSwitch every example to unordered mode
4. DShip it, because the answers are fine

Why: Step 7 requires inspecting trajectory drops, because a plausible reply can hide an unsafe path.

Sourced from Docs by LangChain (MCP)

1. Astrict
2. Bunordered
3. Csubset
4. DsupersetCorrect answer

Why: Superset mode requires the agent's trajectory to contain at least the reference tool calls, and it allows additional ones.

Sourced from Docs by LangChain (MCP)

1. ALANGSMITH_EVAL_MODE
2. BLANGSMITH_TRACINGCorrect answer
3. CLANGCHAIN_DEBUG
4. DOTEL_EXPORT_ONLY

Why: The reference notes name LANGSMITH_TRACING as the switch that enables tracing.

Sourced from Docs by LangChain (MCP)

1. AStrict
2. BPairwise
3. CSuperset
4. DSubsetCorrect answer

Why: Subset mode passes when the agent's calls are contained within the reference trajectory.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai
2. Bcreate_trajectory_llm_as_judgeCorrect answer
3. Cevaluate
4. Dcreate_trajectory_match_evaluator

Why: create_trajectory_llm_as_judge is the agentevals trajectory LLM-as-judge evaluator.

Sourced from Docs by LangChain (MCP)

1. Acreate_pairwise_queue
2. Bcreate_trajectory_llm_as_judge
3. Ccreate_trajectory_match_evaluatorCorrect answer
4. Dtraceable

Why: create_trajectory_match_evaluator performs deterministic matching with a chosen mode.

Sourced from Docs by LangChain (MCP)

1. ASampling rate
2. BWebhooks
3. CCaching
4. DRepetitionsCorrect answer

Why: Repetitions run each example several times, which reveals paths that appear only occasionally.

Sourced from Docs by LangChain (MCP)

1. AAn online sampling filter
2. BA pairwise annotation queue
3. CAn LLM-as-judge on tone
4. DA code evaluatorCorrect answer

Why: Hard, exact rules belong in deterministic code evaluators.

Sourced from Docs by LangChain (MCP)

1. APairwise annotation queue
2. BAutomation rule queue
3. CCI queue
4. DSingle-run annotation queueCorrect answer

Why: Single-run queues support rubrics, assertions and export to datasets.

Sourced from Docs by LangChain (MCP)

1. APrompt tags
2. BDataset splits
3. CRepetitions and caching
4. DFilters and sampling ratesCorrect answer

Why: Online evaluators run on production runs or threads, with filters and sampling rates controlling volume.

Sourced from Docs by LangChain (MCP)

1. ANever; CI only reports
2. BWhen any LLM-judge score changes slightly
3. CWhen critical-path scores regressCorrect answer
4. DOnly when the agent's tests fail to compile

Why: Step 8 blocks merges when critical-path scores regress.

Sourced from Docs by LangChain (MCP)

1. AA trajectory LLM judge with the reference trajectory, plus hard code checks for safety orderingCorrect answer
2. BNo trajectory evaluation; check the answer only
3. CUnordered match only
4. DStrict match only

Why: The LLM judge handles valid variation, while deterministic checks still enforce unsafe-order rules.

Sourced from Docs by LangChain (MCP)

1. AStore real customer data in mock tool responses
2. BShare annotation queues publicly
3. CRun online evaluators on every trace to extend retention
4. DMask sensitive fields before tracing, and scrub exported datasetsCorrect answer

Why: Minimising PII in traces and datasets limits exposure. Extending retention on every trace would increase it.

Sourced from Docs by LangChain (MCP)

1. AThreads queue
2. BSingle-run queue
3. CPairwise queueCorrect answer
4. DOnline evaluator queue

Why: Pairwise annotation queues are built to compare exactly two experiments, while single-run queues review individual runs with rubrics.

Sourced from Docs by LangChain (MCP)

1. AIt is upgraded to extended retentionCorrect answer
2. BIt is moved to a new project
3. CIt is deleted after scoring
4. DIt is copied into a dataset automatically

Why: Running an online evaluator on a trace upgrades that trace to extended retention, which is one reason to keep sampling rates deliberate.

Sourced from Docs by LangChain (MCP)

1. AThe @pytest.mark.langsmith marker
2. BThe @traceable decoratorCorrect answer
3. CAn annotation queue
4. DA pairwise evaluator

Why: The @traceable decorator traces functions that are not covered by a native integration.

Sourced from Docs by LangChain (MCP)

1. AThreadsCorrect answer
2. BPrompt tags
3. CDatasets
4. DExperiments

Why: Threads group multi-turn conversations, while projects group runs and traces.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai
2. BAutomation rulesCorrect answer
3. CThe @traceable decorator
4. DPrompt Hub commits

Why: Automation rules, together with webhooks, act on online evaluation results.

Sourced from Docs by LangChain (MCP)

1. AOnly through Vitest
2. BOnly through annotation queues
3. COnly through manual OTLP spans
4. DNatively, once tracing is enabledCorrect answer

Why: The workflow notes that LangGraph is traced natively.

Sourced from Docs by LangChain (MCP)

1. AIn the Prompt Hub, using commits and tagsCorrect answer
2. BIn the threads view
3. CIn webhook payloads
4. DIn annotation queue rubrics

Why: Prompts are versioned and managed in the Prompt Hub and Playground with commits and tags.

Sourced from Docs by LangChain (MCP)

1. AHuman-labelled casesCorrect answer
2. BThe agent's own unreviewed outputs
3. CWebhook logs
4. DRandomly generated text

Why: Step 5 specifies few-shot examples taken from human-labelled cases.

Sourced from Docs by LangChain (MCP)

1. AOnly prompt tags
2. BOnly model weights
3. CInputs and reference outputsCorrect answer
4. DOnly online evaluator scores

Why: Datasets consist of examples with inputs and reference outputs.

Sourced from Docs by LangChain (MCP)

1. AIgnore it or compare it with a custom rule, while matching order ID and amount exactlyCorrect answer
2. BIgnore all arguments for the refund tool
3. CFail the example whenever the timestamp is present
4. DMatch it exactly

Why: Volatile fields should not cause false failures, but safety-relevant arguments must still match exactly.

Sourced from Docs by LangChain (MCP)

1. A203
2. B600Correct answer
3. C66
4. D200

Why: Each of the 200 examples runs 3 times, giving 600 agent runs.

Sourced from Docs by LangChain (MCP)

1. ADeleting old traces
2. BRotating API keys
3. CRegression testing against a baselineCorrect answer
4. DTraining the judge model

Why: Experiments are compared for benchmarking, unit tests, regression tests and backtesting.

Sourced from Docs by LangChain (MCP)

1. ACypress and Playwright
2. BVitest and JestCorrect answer
3. CAva and Tape
4. DMocha and Karma

Why: The notes list Vitest/Jest integrations for turning test files into experiments.

Sourced from Docs by LangChain (MCP)

1. AIt wraps an OpenAI client so its calls are tracedCorrect answer
2. BIt creates a pairwise queue
3. CIt exports a dataset to CSV
4. DIt mocks the refund API

Why: Wrappers such as wrap_openai trace direct model client calls.

Sourced from Docs by LangChain (MCP)

1. ADuplicate the dataset
2. BUse pairwise queues
3. CEnable repetitions
4. DFan them out through an OpenTelemetry collectorCorrect answer

Why: LangSmith supports OTLP natively, and a collector can fan spans out to several destinations.

Sourced from Docs by LangChain (MCP)

1. ARaw model training checkpointsCorrect answer
2. BCurated cases
3. CProduction traces
4. DSynthetic data

Why: Examples come from curated cases, production traces or synthetic data.

Sourced from Docs by LangChain (MCP)