Online evaluation & production alerts

I'd tier them by cost and signal. Deterministic code evaluators, like schema validity, PII regexes or tool-call errors, are nearly free, so they run on a high share of root runs. LLM-as-judge evaluators cost a model call each, and in LangSmith every evaluated trace is upgraded to extended retention, so I start with a small sample sized to give a stable daily average for the traffic volume. I raise the sample temporarily after a prompt or model deploy, or for a risky segment like a new tenant, using filters on metadata. I also set the sample with alert windows in mind: if a window only contains a handful of scored runs, the alert will be noise.

Sourced from Docs by LangChain (MCP)

I would instrument the app with LANGSMITH_TRACING plus @traceable or wrap_openai, passing a stable thread id in metadata. Traces go to a dedicated prod project, separate from staging, tagged with prompt tag, model, tenant and user tier. On that project I attach run-level online evaluators: cheap code rules on a broad slice and LLM-as-judge on a small sample. Thread-level evaluators score completed conversations. Dashboards show trace count, error rate, latency percentiles, tokens and cost next to feedback scores, grouped by prompt tag and model. Alerts on those metrics page on-call through PagerDuty or Slack webhooks. Automation rules route low scores and negative feedback to annotation queues. Reviewers export confirmed failures to a regression dataset, and prompt fixes are gated by evaluate() before a new Prompt Hub tag is promoted.

Sourced from Docs by LangChain (MCP)

I work it out as a formula and avoid guessing. Judge cost is roughly matching runs per day times the sampling rate, times the number of judge evaluators, times tokens per judge call, times the judge model price. Tokens per call include the judge prompt, any few-shot examples and the run's input and output, which can be large for RAG context. Filters shrink the first term, for example by judging only root runs. I also add the extended-retention cost, because every evaluated trace is upgraded. Then I set sampling rates so the total fits the budget, and I watch it on the dashboard. Code rules carry the broad coverage, so a judge sample can stay small and still be informative.

Sourced from Docs by LangChain (MCP)

I calibrate it against humans before it drives alerts. I take a set of production runs, have reviewers label them in an annotation queue with the same rubric, and compare their labels with the judge's scores. Where they disagree, I refine the judge prompt, add few-shot examples taken from the corrections, or tighten the rubric into clearer criteria. The judge prompt and model are versioned so a score change can be traced to a judge change and not a product change. After launch I keep a small human-review stream on randomly sampled runs to detect drift. Prebuilt templates are a starting point, not a guarantee.

Sourced from Docs by LangChain (MCP)

I reproduce the failure in the Playground with examples from the regression dataset and edit the prompt there, so the change is a new commit in the Prompt Hub. I run evaluate() against the regression dataset, using repetitions if outputs vary, along with the broader benchmark dataset to check that nothing else broke. I compare that experiment with the current production version on the same evaluators. Only if it wins do I move the production tag to the new commit. The app pulls the prompt by tag, so the rollout needs no code deploy, and rollback simply moves the tag back. Afterwards I watch dashboards grouped by prompt tag to confirm the fix in production.

Sourced from Docs by LangChain (MCP)

Many real failures are invisible turn by turn. Each answer might look fine on its own, yet the user rephrases the same question four times, the agent loops on a tool, or the conversation ends without the goal resolved. A thread evaluator sees the whole conversation and can judge resolution, frustration or repeated clarifications. To make it work, the app must pass a consistent thread or session id so LangSmith groups the runs. Then I pair thread scores with run scores. The run scores tell me which turn broke, and the thread score tells me whether the user actually got what they came for, which is closer to the business outcome.

Sourced from Docs by LangChain (MCP)

The evaluator writes feedback onto the run, and an automation rule filtering on a low score picks it up. The rule adds the run to an annotation queue and can also fire a webhook. If many runs fail in a window, the alert on average score or error rate pages on-call through PagerDuty or Slack. A reviewer works the queue with a rubric, confirms or overturns the judge, writes the correct output and exports it to a regression dataset. Engineers then change the prompt in the Playground, run evaluate() against that dataset and compare it with the production experiment. If it wins, they tag the new commit in the Prompt Hub. That way a production failure becomes a permanent test.

Sourced from Docs by LangChain (MCP)

Online evaluators, dashboards, alerts and automation rules are all configured per project. If staging traffic mixes in, test prompts, synthetic users and load tests distort error rates, latency percentiles and average scores. That can cause false alerts or hide a real regression. Judge budget also gets spent on traffic nobody cares about. A per-environment project such as prod-support-bot keeps every metric tied to real users. It also makes access control simpler, because production data often carries personal information and only some people should see it. Staging can still have its own lighter evaluators so problems are caught before release, but the two signal streams never contaminate each other.

Sourced from Docs by LangChain (MCP)

At minimum I attach the thread or session id, prompt tag or commit, model name, app version, tenant and user tier. A request or route identifier also helps. Each field becomes a filter or a grouping key. Dashboards grouped by prompt tag show whether a new commit lowered the groundedness score. Grouping by model separates cost and latency changes after a provider switch. Tenant and tier let evaluator filters spend more judge budget on high-value segments and let alerts fire per segment instead of averaging problems away. Automation rules can also use metadata, for example routing only enterprise-tier failures to a priority queue. Without this metadata you can see that quality dropped, but not why or for whom.

Sourced from Docs by LangChain (MCP)

I use code rules whenever the property can be checked deterministically. Good examples are JSON or schema validity, a PII regex, refusal phrases, required citations, response length or forbidden tool names. They cost almost nothing, are fast and never drift, so they can run at a high sampling rate. LLM-as-judge is for semantic properties a rule cannot capture, such as groundedness against retrieved context, helpfulness or tone. Those judges are slower, cost money per call and need calibration against humans. The usual design layers them. Code rules cover almost everything, while judges run on a sample or on a filtered subset, for example runs a code rule already marked suspicious.

Sourced from Docs by LangChain (MCP)

Threads are built from runs that share the same thread or session id in metadata. A conversation rarely sends an explicit end event, so the evaluator is set to score a thread once it is considered complete, usually after a configured period with no new turns. That timing matters. If it is too short, you score half-finished conversations and wrongly flag the goal as unresolved. If it is too long, alerts on resolution rate lag behind reality. I choose the window from how users actually behave, and I make sure the app passes the same id on every turn. Otherwise one conversation splits into several threads and every score is misleading.

Sourced from Docs by LangChain (MCP)

I alert on a few metrics tied to user impact: error rate, a latency percentile, average judge score and cost, each measured over a window. A window smooths single bad runs. Thresholds come from observed baselines, not round numbers. For sampled judge scores, the window must hold enough scored runs to be meaningful. Alerts are scoped with filters, such as per prompt tag or tier, so a small segment's regression is not averaged away. Every notification links to the filtered runs, so on-call can triage quickly. I split severity: pages for errors and outages, Slack for slower quality drift. Then I review which alerts were acted on and retune the rest.

Sourced from Docs by LangChain (MCP)

I start with the built-in project metrics: trace count for traffic, error rate for reliability, latency percentiles such as p50 and p99 for user experience, and token usage and cost for spend. Next to them I place feedback scores from online evaluators, such as JSON validity rate, groundedness, helpfulness and thread-level resolution, plus explicit user feedback. Every chart can be grouped by metadata such as model, prompt tag or tenant. That grouping turns a dashboard from a status page into a diagnostic tool. A cost jump grouped by model, for example, points straight at a routing change. I also chart the evaluators themselves, including how many runs they scored, so a silent evaluator is noticed.

Sourced from Docs by LangChain (MCP)

Cost and token usage are built-in project metrics, so I chart them over time grouped by model and prompt tag, and I alert when cost over a window crosses a baseline-derived threshold. Common causes include a prompt commit that adds long instructions, a retrieval change that stuffs more context, an agent loop making extra tool calls, or a fallback to a pricier model. Grouping by prompt tag shows which release caused it. Grouping by tenant shows whether one customer's usage pattern did. A code rule can flag individual runs whose tokens exceed a limit and route them to review. Judge spend should be tracked as well, since evaluators add their own cost.

Sourced from Docs by LangChain (MCP)

Sampling gives an estimate, so I make sure each alert window contains enough scored runs for the average to be stable. For low-traffic segments that may mean a longer window or a higher sampling rate for that segment's filter. I combine signals: broad code rules, explicit user feedback and error rates all cover far more traffic than the judge. After a deployment I temporarily raise sampling on runs with the new prompt tag, then lower it once confidence is established. Grouping by prompt tag also compares old and new versions side by side. Before release, the offline regression suite catches known failure modes, so online sampling mainly needs to find new ones.

Sourced from Docs by LangChain (MCP)

Production traces may contain personal data, so I minimise it at the source. Sensitive fields are redacted or masked before they are traced, and only the fields the evaluator needs are logged. The judge's model provider must be approved under the same data agreements as the main app. Annotation queue access is restricted to trained reviewers, and workspace roles limit who can open the prod project. A PII regex code rule can flag leaks and route them for cleanup. I also remember that evaluated traces move to extended retention, so the retention policy and deletion obligations must cover them. Regression datasets built from production need the same review before they are shared.

Sourced from Docs by LangChain (MCP)

I control volume and prioritise. The filters on automation rules should be precise. Routing every low score is often too much, so I combine conditions, for example a low judge score plus negative user feedback, or an errored tool call on a paid tier. I keep separate queues by severity or domain, so urgent cases are not buried. Reservations stop two reviewers working the same run, and reviewer counts let me require more than one opinion only where it matters. A clear rubric keeps reviews fast and consistent. If backlog still grows, I sample within the rule's filter and look for clusters, since many flags usually come from one root cause.

Sourced from Docs by LangChain (MCP)

In the single-run annotation queue, reviewers score the run against the rubric, confirm it is truly a failure, and correct the output so the example has a trustworthy reference. Confirmed cases are exported to a regression dataset as inputs plus the corrected reference output. I keep metadata such as failure category and source date, so the dataset can be sliced. Near-duplicates are pruned so one incident does not dominate. Then any prompt or model change runs evaluate() against that dataset and is compared with the current production experiment. Over time the dataset records every bug users actually hit, which makes it much more valuable than synthetic cases alone.

Sourced from Docs by LangChain (MCP)

The run simply gets no feedback score from that evaluator. The danger is silence: an average score computed over fewer runs looks healthy, or the chart flattens and nobody notices. So I monitor evaluator coverage, meaning how many runs each evaluator scored in a window, and alert if it drops. Judge failures often come from rate limits, a changed model name, a revoked API key or oversized inputs, so I keep judge prompts within context limits. I treat evaluators as production code: versioned, tested on a few examples before enabling, and owned by a team. Missing scores should never be read as passing scores.

Sourced from Docs by LangChain (MCP)

The app pulls prompts by tag, and each run records which prompt tag or commit served it. During a rollout, part of the traffic uses the candidate. I group dashboards by prompt tag to compare error rate, latency, cost and judge scores between old and new on live traffic. I temporarily raise the sampling rate for runs with the candidate tag, so its scores reach a stable sample quickly. Alerts filtered to that tag give early warning. If the candidate is worse, I move the tag back, and no code deploy is needed. Its failing runs are routed to review and added to the regression dataset for the next attempt.

Sourced from Docs by LangChain (MCP)

Offline evaluation runs evaluate() on fixed datasets with reference outputs, so it is repeatable and comparable. It suits gating releases, regression tests and backtesting, and pytest or Vitest integrations put it in CI. But it only covers cases you thought of. Online evaluation scores real production runs and threads, so it catches new user behaviour, data drift and failures nobody anticipated. It usually has no reference output, and it is sampled. Each feeds the other: online failures, after human review, become offline regression examples, and offline results set the baselines for online alert thresholds. Relying only on offline misses drift. Relying only on online means shipping regressions before you find them.

Sourced from Docs by LangChain (MCP)

Multi-turn conversations no longer group into threads. Each turn appears as an unrelated trace, so thread-level evaluators either have nothing to score or score fragments. Goal resolution and frustration detection become meaningless, because they depend on seeing the whole conversation. Debugging also gets harder, since a reviewer cannot see what the user asked three turns earlier. A subtler problem is ids that change mid-session, for example after a reconnect or a server-side regeneration, which split one conversation into two. I would generate the id once per session on the client or gateway, persist it, and add a code-rule check or dashboard metric for runs missing it.

Sourced from Docs by LangChain (MCP)

There are three. First, cost: every evaluated trace is stored longer, so a high sampling rate raises storage spend as well as judge spend. Second, privacy: personal data in those traces lives longer, which has to match your retention and deletion commitments. Third, a benefit: the runs you scored, which are often the interesting ones, remain available for later review, dataset export and debugging. So when I set sampling rates and filters, I count retention as part of the budget. Automation rules can also extend retention deliberately for flagged runs, which is often cheaper than evaluating more broadly just to keep evidence.

Sourced from Docs by LangChain (MCP)

Pairwise queues compare exactly two experiments, so I use them when absolute scores do not decide the question. A typical case is choosing between a current prompt and a candidate whose judge scores are close. Reviewers see both outputs for the same input and pick the better one, which is often easier and more consistent than scoring each on an absolute scale. That helps with subjective qualities such as tone or helpfulness. Pairwise LLM-as-judge evaluators can do something similar automatically, but a human pairwise pass is a good check before promoting a prompt tag on a sensitive change. For triaging single production failures, the single-run queue is the right tool.

Sourced from Docs by LangChain (MCP)

I assume webhooks can be delayed, duplicated or lost. The receiver, such as a Slack relay or ticketing service, should be idempotent, keyed on the run id, so duplicate deliveries do not create duplicate tickets. It should respond quickly and do heavy work asynchronously. It must authenticate requests with a shared secret or a signed header, and it should not leak trace contents into channels with wide access. Send a link to the run instead of the full payload where possible. For anything critical, the webhook is not the only path. Dashboards and annotation queues still hold the data, so a missed notification does not mean a lost failure.

Sourced from Docs by LangChain (MCP)

LangSmith accepts traces natively over an OTLP endpoint, so an app already instrumented with OpenTelemetry can send spans without rewriting its tracing code. A collector adds fan-out: the same spans can go to LangSmith for LLM-specific evaluation and to an existing observability backend for infrastructure monitoring. It is also a central place to drop noisy spans, add resource attributes such as environment, and redact sensitive fields before data leaves your network. The trade-off is one more component to run and scale. A misconfigured collector can silently drop traces, so I monitor trace counts on the LangSmith dashboard to catch gaps.

Sourced from Docs by LangChain (MCP)

Agents fail in their trajectory, not only in the final answer. Online, I use code rules on child runs, for example flagging errored tool calls, too many steps or forbidden tools. An LLM-as-judge scores the final answer on a sample of root runs. Thread-level evaluators check whether the user's goal was resolved across turns. Automation rules route errored tool calls to review. Offline, the agentevals package provides trajectory evaluators, such as a trajectory LLM-as-judge, that compare the agent's sequence of steps against expectations. So problematic production trajectories go into a dataset and are scored with those evaluators when the agent's prompt or tools change.

Sourced from Docs by LangChain (MCP)

Every run carries a tenant id in metadata, so evaluators, dashboards and alerts can be filtered or grouped by tenant. I can give high-value or regulated tenants a higher judge sampling rate through separate filters, while still running cheap code rules everywhere. Per-tenant charts stop a regression in one tenant's data or configuration from being averaged away by others. Privacy matters: some tenants may forbid sending their data to a third-party judge or into datasets. Their filters should exclude them from LLM judges and from automation rules that export to datasets. Reviewer access should respect tenant boundaries, and exports need checks so one customer's data does not appear in shared test sets.

Sourced from Docs by LangChain (MCP)

If automation only routes judge-flagged runs, the dataset reflects the judge's blind spots. Failures the judge misses never get reviewed. I feed the queues from several sources: low judge scores, explicit negative user feedback, errored tool calls, code-rule hits, and a small random sample of runs that looked fine. Reviewer findings on that random stream show the judge's false negatives and improve its prompt or few-shot examples. I also label failure categories, so I can see whether some types are overrepresented. Finally, curated and synthetic cases fill gaps for rare but important scenarios that production traffic has not produced yet.

Sourced from Docs by LangChain (MCP)

I pin the judge model version and keep the judge prompt versioned, so scores are only compared across a stable instrument. If I need to change the judge, I run old and new judges on the same labelled set, check agreement with human labels, and mark the switch on the dashboard so a step change is not mistaken for a product regression. Ideally I run both briefly in parallel. I avoid having the judge share the exact model and prompt style of the system under test, where it may favour its own outputs. And a human-reviewed sample stays in place as ground truth.

Sourced from Docs by LangChain (MCP)

I open the linked filtered runs and first check whether the alert is real. Did the evaluator's sample size drop, did the judge change, or is it a single tenant? Then I group by metadata: prompt tag, model, tenant and app version. A drop isolated to a new prompt tag means rolling the tag back. A drop tied to a retrieval or upstream data change shows up as low scores with weak retrieved context. I read a handful of the lowest-scoring runs to confirm the failure is genuine. After mitigation, I make sure those runs are in an annotation queue, so confirmed cases reach the regression dataset and a postmortem can follow.

Sourced from Docs by LangChain (MCP)

LLM latency is heavily skewed. Most requests may be quick while a tail of agent loops, long generations or retries takes much longer. An average hides that tail and can stay flat while a growing share of users waits a long time. Percentiles such as p95 or p99 show the worst experiences directly, and the dashboard exposes them as built-in metrics. I usually alert on a high percentile over a window and keep p50 on the chart for context. Grouping by model or prompt tag then shows whether a provider slowdown or a prompt producing longer outputs caused it. Sometimes latency regressions are really cost regressions too.

Sourced from Docs by LangChain (MCP)

User feedback, such as thumbs up or down or a correction, is attached to runs as feedback alongside evaluator scores, so both appear on dashboards and in filters. It covers every run a user rates without judge cost, but it is sparse and skewed toward strong reactions. I use it in three ways. First, as an automation rule trigger: negative feedback sends the run to the annotation queue. Second, as calibration: if users dislike runs the judge rated highly, the judge needs work. Third, as an alert signal over a window, alongside judge averages. Agreement between the two raises confidence, and disagreement shows exactly where human review should focus.

Sourced from Docs by LangChain (MCP)

1. ALANGCHAIN_DEBUG
2. BOTEL_SDK_DISABLED
3. CLANGSMITH_TRACINGCorrect answer
4. DLANGSMITH_EVAL_MODE

Why: The reference notes name LANGSMITH_TRACING as the switch, used together with @traceable or wrappers.

Sourced from Docs by LangChain (MCP)

1. AA human annotation queue
2. BA code rule checking JSON validityCorrect answer
3. CAn LLM-as-judge for helpfulness
4. DA pairwise LLM judge

Why: Deterministic code rules cost almost nothing per run, unlike judge model calls or human time.

Sourced from Docs by LangChain (MCP)

1. ATrigger a webhook
2. BAutomatically rewrite and redeploy the promptCorrect answer
3. CAdd the run to an annotation queue
4. DAdd the run to a dataset

Why: Rules can queue, add to datasets, trigger webhooks or extend retention, while prompt fixes go through the Playground and evaluate().

Sourced from Docs by LangChain (MCP)

1. AThe trace is deleted after scoring
2. BThe trace is upgraded to extended retentionCorrect answer
3. CThe trace is copied to every project
4. DThe trace is automatically added to a dataset

Why: LangSmith upgrades a trace to extended retention when an online evaluator runs on it, so higher sampling rates also raise retention costs.

Sourced from Docs by LangChain (MCP)

1. APrompt Hub tags
2. BPairwise annotation queue
3. Cpytest integration
4. DAutomation rule that adds matching runs to an annotation queueCorrect answer

Why: Automation rules apply a filter, such as a low feedback score, and take an action like adding the run to an annotation queue. Pairwise queues compare exactly two experiments, so they do not fit this job.

Sourced from Docs by LangChain (MCP)

1. Awith_feedback
2. Bwrap_openaiCorrect answer
3. Ctrace_llm
4. Dopenai_monitor

Why: wrap_openai wraps the OpenAI client so its calls are traced automatically.

Sourced from Docs by LangChain (MCP)

1. ADatasets, built from curated examples
2. BExperiments, created by evaluate()
3. CAnnotation queues, through reservations
4. DThreads, built from a shared thread or session id in metadataCorrect answer

Why: Threads group multi-turn conversations, which requires a stable thread or session id on each run.

Sourced from Docs by LangChain (MCP)

1. ASo dashboards, evaluators and alerts only see real trafficCorrect answer
2. BBecause staging cannot be traced
3. CBecause online evaluators only work in staging
4. DTo make prompts unversioned in production

Why: Evaluators and dashboards are attached per project, so mixing environments pollutes the signal.

Sourced from Docs by LangChain (MCP)

1. AThe prompt commit was untagged
2. BThe user's goal was never resolved across the conversationCorrect answer
3. CA single response was invalid JSON
4. DA tool call returned an HTTP error

Why: Goal resolution and frustration emerge across turns, and single-turn judges miss them.

Sourced from Docs by LangChain (MCP)

1. AFrustration detected
2. BRubric pass rate
3. CLatency percentilesCorrect answer
4. DGroundedness score

Why: Trace count, error rate, latency percentiles, tokens and cost are built in, while the others come from evaluators.

Sourced from Docs by LangChain (MCP)

1. AGroup by dataset split
2. BGroup by annotation queue
3. CGroup by the model metadata fieldCorrect answer
4. DGroup by reviewer name

Why: Runs carry model metadata, so grouping by model separates cost per model.

Sourced from Docs by LangChain (MCP)

1. AA link back to the filtered runs that breached the thresholdCorrect answer
2. BThe full judge prompt text
3. CA copy of the regression dataset
4. DThe Prompt Hub commit history

Why: The workflow links alerts to the filtered runs so on-call can inspect the cases directly.

Sourced from Docs by LangChain (MCP)

1. ATwo reviewers working the same run at onceCorrect answer
2. BRuns from being exported to datasets
3. CJudges from scoring threads
4. DAlerts from firing twice

Why: Reservations stop two people grabbing the same run.

Sourced from Docs by LangChain (MCP)

1. AIt copies the prompt from the alert payload
2. BIt loads the judge prompt
3. CIt pulls the prompt from the Prompt Hub by tagCorrect answer
4. DIt reads the latest dataset example

Why: The winning commit is tagged in the Prompt Hub, and the app pulls by tag.

Sourced from Docs by LangChain (MCP)

1. AClear the annotation queue
2. BMove the production tag back to the previous commitCorrect answer
3. CDelete the production project
4. DDisable tracing

Why: Because the app pulls by tag, moving the tag reverts behaviour without a deploy.

Sourced from Docs by LangChain (MCP)

1. Atraceable()
2. Bwrap_openai()
3. Cpull_tag()
4. Devaluate()Correct answer

Why: evaluate() runs experiments, with options like repetitions, concurrency and caching.

Sourced from Docs by LangChain (MCP)

1. ARun evaluators on staging traffic too
2. BLower the judge sampling rate or narrow its filter, and keep cheap code rules broadCorrect answer
3. CRun the judge on every child run
4. DRemove all code rules

Why: Sampling and filters bound judge calls, while code rules keep coverage cheaply.

Sourced from Docs by LangChain (MCP)

1. AMatching runs per day × sampling rate × number of judge evaluatorsCorrect answer
2. BTotal projects × dataset size
3. CNumber of reviewers × rubric items
4. DPrompt commits × tags

Why: Only runs passing the filter and sampled are scored, once per judge evaluator.

Sourced from Docs by LangChain (MCP)

1. AThe thread ids are stable
2. BThe evaluator is failing, for example from rate limits or a bad keyCorrect answer
3. CQuality has improved
4. DTraffic moved to staging by design

Why: Falling coverage signals evaluator failure, and missing scores must not be read as passes.

Sourced from Docs by LangChain (MCP)

1. ASo retention is shortened
2. BSo the thread id changes
3. CSo the example has a trustworthy reference outputCorrect answer
4. DSo the judge can skip it

Why: Dataset examples pair inputs with reference outputs that future experiments are checked against.

Sourced from Docs by LangChain (MCP)

1. AReviewer count and reservations
2. BCommits and tags
3. CRepetitions and concurrency
4. DFilters and sampling rateCorrect answer

Why: Online evaluators are configured with filters that select runs and a sampling rate. Repetitions and concurrency belong to offline evaluate(), and reviewer counts belong to annotation queues.

Sourced from Docs by LangChain (MCP)

1. AOnly email digests
2. BOnly the Playground
3. CPrompt Hub tags
4. DPagerDuty or a webhook to SlackCorrect answer

Why: Step 6 routes breaches to PagerDuty or a Slack webhook.

Sourced from Docs by LangChain (MCP)

1. AExactly twoCorrect answer
2. BAt least three
3. CAny number
4. DOnly one, against references

Why: Pairwise queues work on exactly two experiments.

Sourced from Docs by LangChain (MCP)

1. ATo reduce judge cost
2. BTo bypass sampling rates
3. CTo extend retention
4. DTo account for non-deterministic outputs before comparingCorrect answer

Why: Repeating runs shows variance, so a comparison is not decided by one lucky output.

Sourced from Docs by LangChain (MCP)

1. A@pytest.mark.trace
2. B@pytest.mark.online_eval
3. C@pytest.mark.judge
4. D@pytest.mark.langsmithCorrect answer

Why: The notes list @pytest.mark.langsmith as the pytest integration.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai
2. BPrompt Hub
3. COTLP collector
4. DagentevalsCorrect answer

Why: agentevals supplies trajectory evaluators such as trajectory LLM-as-judge.

Sourced from Docs by LangChain (MCP)

1. AAutomatic LLM-as-judge scoring
2. BAnnotation queue reservations
3. CPrompt version tags
4. DFan-out of the same spans to multiple backendsCorrect answer

Why: The notes describe native OTLP ingest with fan-out via a collector.

Sourced from Docs by LangChain (MCP)

1. AWindows avoid extended retention
2. BWindows disable sampling
3. CA window smooths single outliers and holds enough sampled scores to be meaningfulCorrect answer
4. DIndividual runs cannot be scored

Why: Averaging over a window reduces noise from sparse sampled scores.

Sourced from Docs by LangChain (MCP)

1. ARoute only staging runs
2. BAlso route negative user feedback and a random sample of passing runs to reviewCorrect answer
3. CRoute only runs the judge scored low
4. DRoute nothing until a quarterly review

Why: Reviewing runs the judge passed reveals its false negatives.

Sourced from Docs by LangChain (MCP)

1. AA dashboard refresh
2. BAn errored tool callCorrect answer
3. CA successful prompt tag pull
4. DA completed dataset export

Why: The workflow lists errored tool calls as a rule filter alongside low scores and negative feedback.

Sourced from Docs by LangChain (MCP)

1. ADisable thread ids
2. BRedact or mask sensitive fields, and use only approved judge providersCorrect answer
3. CRaise the sampling rate
4. DMerge prod and staging projects

Why: Production traces may contain personal data, so they should be minimised before evaluation and review.

Sourced from Docs by LangChain (MCP)

1. ARaising the alert threshold
2. BDeleting the run
3. CAn idempotent receiver keyed on the run idCorrect answer
4. DTurning off automation rules

Why: Deliveries can repeat, so deduplicating by run id prevents duplicate tickets.

Sourced from Docs by LangChain (MCP)

1. AThe judge temperature
2. BThe dataset name
3. CThe prompt tag or commit on each runCorrect answer
4. DThe reviewer id

Why: Grouping dashboards and evaluator filters by prompt tag separates old and new versions.

Sourced from Docs by LangChain (MCP)