Cost & latency monitoring from traces

First I'd open the dashboard for that feature and check whether p99 latency rose for all models or one, and whether it lines up with a deploy or prompt version in the metadata. Then I'd filter the project to slow runs for that feature in the window and open a few traces. The trace tree shows latency per child run, so I can see if the time is in retrieval, a tool call, retries, or the agent taking extra turns. I'd compare against fast traces from before the change. If the cause is a prompt or model change, I'd add those slow inputs to a dataset and run an experiment comparing old and new versions so the fix is verified, not guessed.

Sourced from Docs by LangChain (MCP)

The app enables tracing with LANGSMITH_TRACING and an API key, and each environment writes to its own project. Entry points and key steps such as retrieval, tools and LLM calls are decorated with @traceable, and model clients are wrapped with wrap_openai or covered by the LangChain or LangGraph integration so token counts are captured. Custom model calls return usage_metadata plus ls_provider and ls_model_name so cost comes from the pricing map. Services already on OpenTelemetry send spans to the OTLP endpoint through a collector that also feeds the APM. Runs nest into traces, and threads group conversations. On top of that I build dashboards grouped by model and feature, alerts that go to a webhook or pager, and automation rules that push slow or expensive runs into datasets for offline experiments.

Sourced from Docs by LangChain (MCP)

I attach feature, tenant tier, prompt version and model at minimum, plus environment through the project itself. Feature lets me group cost and latency per product surface, so a spike points to one feature and not the whole app. Tenant tier shows whether premium or free traffic drives cost and helps with capacity and pricing decisions. Prompt version ties a regression to a specific prompt commit or tag. Model separates the effect of model swaps from prompt changes. I keep keys consistent and low-cardinality, documented in one shared constants module. If teams invent their own spellings like feature and feature_name, the dashboards split silently and the numbers get misread. Raw user IDs stay out of grouping keys.

Sourced from Docs by LangChain (MCP)

Cost is derived from token usage on the run and the model pricing map. Wrapped clients such as wrap_openai, and the LangChain integrations, record token counts and model identity automatically. For custom or self-hosted calls, the run must return usage_metadata and set ls_provider and ls_model_name so the right price entry can be matched. Cost goes wrong when usage is missing, which gives zero cost, or when the model name does not match the pricing entry. It also goes wrong when the pricing map is out of date after a provider price change, or when a fallback model is logged under the requested model's name. I audit for runs with tokens but no cost, and I reconcile totals against provider invoices periodically.

Sourced from Docs by LangChain (MCP)

The feature label must come from the caller, not from the helper. I set feature metadata at each entry point's root run and propagate it through the tracing context, so the helper's child runs carry whichever feature invoked them. I never hardcode a feature inside the shared function. Grouping by feature should then aggregate across the whole trace, not only the helper's own runs. I verify this with a test: call each feature with a known request, then check in LangSmith that the helper's runs carry the correct feature value. I also add a check that flags LLM runs with a missing feature key, because untagged runs would otherwise land in an 'unknown' bucket or skew one feature's numbers.

Sourced from Docs by LangChain (MCP)

They measure different user experiences and have different causes. Time to first token is what the user feels as responsiveness. It is driven by retrieval, prompt assembly, queueing at the provider and input length. Total latency is driven mostly by output length and generation speed. A change that makes answers longer raises total latency without hurting perceived speed. A slow retrieval step hurts time to first token directly. If I only charted total latency, I could chase the wrong fix, for example trimming outputs when the real problem is a slow vector search. So I check both in the trace and on dashboards, and I make sure streaming calls are wrapped so the first-token timing is actually captured.

Sourced from Docs by LangChain (MCP)

Averages hide the shape of the distribution. LLM latency is long-tailed, because a few requests with large contexts, retries or extra agent turns can take far longer than typical ones. p50 tells me what most users experience and catches broad regressions such as a slower model. p99 catches tail problems such as a timeout-and-retry pattern that affects a small share of traffic but ruins those sessions. A mean can move because of a handful of outliers or stay flat while the tail gets worse. Grouping both percentiles by model and feature then shows whether the tail belongs to one feature, which is usually where I start digging.

Sourced from Docs by LangChain (MCP)

Cost and latency monitoring needs token counts, timings and metadata, not raw user content. So I redact or hide sensitive inputs and outputs before they leave the app, using the SDK's masking options or redaction in the OTel collector for spans sent over OTLP. I never put secrets, API keys or raw personal identifiers in metadata or tags. Tenant identifiers are mapped to tiers or hashed IDs. API keys are scoped per environment and stored in a secret manager. Access to production projects is restricted to people who need it. I also check that datasets built by automation rules from production traces go through the same redaction, because examples copied from traces can outlive the original retention.

Sourced from Docs by LangChain (MCP)

Cost is computed from token counts and a model pricing map, so both inputs must be right. I use wrappers like wrap_openai or the LangChain integration so token usage is captured automatically. For custom or self-hosted models I return usage_metadata and set ls_provider and ls_model_name, otherwise those runs show no cost. I review the pricing map whenever we change models or the provider changes prices. I also track a metric for the share of LLM runs with missing token data, and I reconcile the monthly total against the provider invoice. If the gap is large, it usually points to an unwrapped code path.

Sourced from Docs by LangChain (MCP)

I'd alert on aggregates over a window, like cost per hour or p99 latency over fifteen minutes, not on individual expensive runs. Thresholds come from a few weeks of baseline data per feature, since a research agent and an autocomplete feature have very different normal costs. Alerts go to a webhook that posts to the team channel, and only sustained, large breaches page on-call. Each alert links to a filtered view of the project so the responder lands on the relevant traces. Finally, I review alerts monthly: anything that fired without action gets tuned or removed, and any incident we missed gets a new rule.

Sourced from Docs by LangChain (MCP)

Projects are the main boundary for runs, dashboards, alerts and automation rules. If dev, staging and production share a project, load tests and debugging sessions inflate production cost and latency charts. Alerts then fire on synthetic traffic, or real spikes get diluted. Separate projects keep production metrics clean, let me apply stricter alert thresholds and sampling to production only, and make retention and access easier to reason about. I set the project per environment through configuration, not code, so a deploy cannot accidentally point staging at the production project. Environment can also go in metadata as a safety net, but the project split is what keeps the dashboards trustworthy.

Sourced from Docs by LangChain (MCP)

I open the trace tree and look at the shape, not just the total time. A retry loop shows repeated sibling runs of the same LLM or tool call, often with errors or timeouts on the earlier attempts. An agent taking extra turns shows more tool-call and LLM-call cycles than the usual trace for that feature. I compare the trace with a normal one for the same feature. To catch this at scale, I chart run count or token usage per trace by feature, and I add an automation rule that samples traces above a token or latency threshold into an annotation queue. Trajectory evaluators from agentevals can then judge whether the extra steps were justified.

Sourced from Docs by LangChain (MCP)

I write a small scheduled job that queries runs through the LangSmith SDK, filtered by project, time window and metadata such as feature or model. It computes the metric, for example cost per feature per hour or tokens per trace, and compares it with a threshold or baseline. If the threshold is breached, it posts to the same webhook or paging tool the native alerts use, so on-call sees one stream. I make the job idempotent and include the window and grouping keys in the payload so duplicates can be suppressed. I also alert if the job itself fails or finds no runs. Silence from a broken monitor should never look like 'everything is fine'.

Sourced from Docs by LangChain (MCP)

A single trace covers one request, but cost in chat products often grows across a conversation, because history is resent and the context gets longer each turn. Threads group the traces of a multi-turn conversation, so I can read total tokens, cost and latency per conversation, not just per message. That shows patterns such as late turns being much more expensive or slower. Those patterns point to fixes like history summarisation or truncation. It also lets me sample whole expensive conversations into review queues, and online evaluators can run at thread level. To make this work, every turn must carry the same thread identifier in its metadata.

Sourced from Docs by LangChain (MCP)

I start from measured numbers in our own system, not assumptions. Requests per day multiplied by runs per trace gives runs per day. Runs per trace depends on how many steps I decorate: retrieval, tools and each LLM call. Payload size per run depends on prompt and output lengths. Agent features usually have more runs per trace and larger payloads than simple completions. I measure these in staging with real traffic patterns, then project for peak load and growth. Extended retention matters too, because online evaluators upgrade the traces they run on. The estimate drives choices about which steps to trace, how much to sample in automation rules, and what my plan and quotas need to cover.

Sourced from Docs by LangChain (MCP)

Tracing must never be on the critical path. The SDK sends runs in the background, so a LangSmith outage should cost me observability, not user requests. I confirm this in a staging test by blocking the endpoint and checking that request latency and error rates stay flat. I keep timeouts conservative and make sure the app does not crash on tracing errors. For OTel paths, the collector buffers and retries independently of the app. The risk is gaps in the data, so dashboards and alerts must not read a gap as low cost. I alert on unusually low run volume, and I treat cost totals from outage windows as incomplete.

Sourced from Docs by LangChain (MCP)

The app exports once and the collector fans the same spans out, so there is one instrumentation path to maintain and no double SDK overhead in the service. Because the spans are identical, the APM and LangSmith share trace IDs. An on-call engineer can jump from a slow HTTP request in the APM to the LLM-level breakdown in LangSmith, with prompts, tokens and cost for the same request. The collector is also a natural place for redaction, batching and retry. The trade-off is that the collector becomes infrastructure I must run and monitor, and I need to confirm that LLM-specific attributes such as model and token usage survive the pipeline.

Sourced from Docs by LangChain (MCP)

First I check coverage: are all model calls traced? Untraced background jobs, scripts or another environment's project can explain a gap. Then I look for runs with tokens but no cost, which usually means a missing or mismatched ls_model_name or an absent pricing entry. Next I compare the pricing map with the provider's current prices and check for price changes during the month. Then I look for token types that may be priced differently, such as cached or reasoning tokens, and check whether they are recorded. Fallback calls logged under the wrong model are another candidate. I fix the attribution, document known gaps, and keep the invoice as the billing source of truth while LangSmith remains the attribution tool.

Sourced from Docs by LangChain (MCP)

Each attempt must be its own run, recorded with the model that actually served it. If wrap_openai or the integration wraps the real client call, the model identity comes from the actual request. For custom fallback code, I set ls_provider and ls_model_name from the model actually used, not from the configured default. The failed primary attempt stays visible as an errored run with its latency, because its time and any billed tokens are real. I also add metadata such as a fallback flag, so dashboards can show fallback rate per feature. Without this, the backup model's cost and latency get charged to the primary, and teams make the wrong model decisions.

Sourced from Docs by LangChain (MCP)

I build a dataset from slow, expensive and representative production runs collected by automation rules, plus curated cases. I run evaluate() against the current configuration and the candidate, with repetitions to smooth out variance. Code evaluators score token usage and latency next to quality evaluators, so I see the trade-off in one comparison. I compare the experiments side by side and set a gate, such as no meaningful increase in tokens per example without a quality gain. One caution: caching is useful for iterating on evaluators, but cached model responses make latency and cost meaningless. So the run that measures cost and latency must call the models for real.

Sourced from Docs by LangChain (MCP)

I start with the highest-value path. I enable tracing to a staging project, wrap the model client so tokens and cost appear immediately, and add @traceable to the entry point. Then I add metadata keys such as feature, model and prompt version, and verify grouping works. Next I decorate retrieval and tool steps so the trace tree shows where time goes. I validate costs against known requests, then turn on production with its own project. After a baseline period I add dashboards by model and feature, then alerts with thresholds tuned to that baseline. Automation rules and offline experiments come last. Each step is verified before the next, so attribution is trusted before anyone acts on it.

Sourced from Docs by LangChain (MCP)

I treat it like code that needs tests. In staging I send known requests for each feature and check the trace tree: every LLM run sits under the right parent step, carries the right feature and model, and has token usage and non-zero cost. I check that latency belongs to the right child, for example that retrieval time is not hidden inside an undecorated parent. In production I run a scheduled audit through the SDK for LLM runs missing usage, model name or feature metadata, and alert when the share rises. Any new model client or helper gets the same checks in code review before it ships.

Sourced from Docs by LangChain (MCP)

I start at the feature-grouped dashboard and split the spike by model, prompt version and tenant tier, to see whether it is volume, tokens per request, or price per token. If volume is flat but tokens per trace rose, I open expensive traces and compare them with older ones. Common culprits are growing conversation history, retrieval returning more or longer chunks, agents taking extra tool turns, or retry loops after provider errors. If the model field changed, a fallback or config flag may have switched models. If one tenant dominates, it may be abuse or a batch integration. I confirm with SDK queries over the window, then add an alert for that pattern.

Sourced from Docs by LangChain (MCP)

The Prompt Hub versions the prompt itself through commits and tags, but production runs need to say which version they actually used. With prompt version in run metadata, I can group dashboards by it and see that a new commit raised tokens per request or p99 latency, for example because it added long instructions or few-shot examples. It also separates prompt effects from model effects when both change around the same time. During a gradual rollout, two versions run side by side, and the metadata lets me compare them on live traffic. I pull the version from the Hub tag or commit at load time, so the label cannot drift from what was sent.

Sourced from Docs by LangChain (MCP)

Running an online evaluator on a trace upgrades that trace to extended retention, so broad evaluators keep far more data around than the default. LLM-as-judge evaluators also make their own model calls, which add cost. That is why I apply filters and sampling rates: target runs that matter, such as slow, expensive or flagged ones, and sample the rest. I also keep evaluator runs separate from product runs in analysis, so judge costs do not get counted as feature cost. When reviewing the monitoring budget, I include evaluator spend and extended-retention volume explicitly, not as a surprise line.

Sourced from Docs by LangChain (MCP)

Embedding calls cost money too, both at query time and during indexing, and they are easy to miss. I trace the query-time embedding step as its own run under the retrieval step, with token usage and correct provider and model fields, so it is priced and shows up in the trace tree. Indexing jobs go to a separate project or carry a distinct feature value, so batch ingestion spikes do not look like a user-facing cost regression. On dashboards I can then split retrieval cost from generation cost per feature. That matters when deciding whether to optimise chunking and re-embedding, or the prompt and model.

Sourced from Docs by LangChain (MCP)

Grouping works best on keys with a bounded set of values: feature, model, tenant tier, prompt version, environment. Raw user IDs, request IDs or free-text values make charts unreadable and hide the signal. I keep high-cardinality identifiers out of grouping, mapping tenants to tiers, and use them only for filtering when investigating a specific case. I define the allowed values in shared code and validate them, for example by rejecting unknown feature names in tests. This also protects attribution, because typos or ad hoc labels would otherwise split one feature's cost across several buckets.

Sourced from Docs by LangChain (MCP)

I route alerts to a paging tool that already handles retries, deduplication and escalation, and I avoid custom endpoints on the critical path where possible. For webhooks I own, the receiver is idempotent and quick to acknowledge, and does heavy work asynchronously. Scheduled-job alerts include the time window and grouping keys, so duplicates can be collapsed. I test the full path periodically with a synthetic alert. I also run a heartbeat check, so a silent failure in the alerting chain, or a monitoring job that stops running, is itself detected and does not look like a quiet day.

Sourced from Docs by LangChain (MCP)

Errors are often the hidden cause of both. A failing call that is retried adds latency and can add billed tokens, so cost rises while successful output stays the same. Timeouts inflate p99. Rate-limit errors can trigger fallbacks to a different, more expensive model. Charting error rate by model and feature next to cost and latency lets me see that a spike in spend lines up with a spike in errors for one provider. Without it, I might blame a prompt change. In the trace tree, errored child runs keep their own latency and usage, so the cost of failure stays visible.

Sourced from Docs by LangChain (MCP)

If the app is built on LangChain or LangGraph, the integration already traces chains, nodes, tool calls and model calls, with token usage and model information, once tracing is enabled. Adding @traceable to the same steps would duplicate runs or create confusing nesting. I use the integration as the base and add @traceable only to custom code outside the framework, such as a bespoke retrieval service or post-processing step. Metadata and tags still need to be passed in through the framework's config, so feature and prompt version reach every run. The goal is one run per real step, not two.

Sourced from Docs by LangChain (MCP)

An APM is good at service-level latency, infrastructure metrics and existing on-call workflows. But it usually lacks LLM-specific context: prompts, token usage per call, model pricing, nested agent steps, threads, and the link to datasets and evaluations. LangSmith gives that context and turns slow or expensive runs into datasets and experiments. Running only LangSmith loses infrastructure context, and running only the APM loses LLM attribution. Sending the same OTel spans to both through a collector, with shared trace IDs, gives each team its natural view. The cost is maintaining the collector and keeping attribute conventions consistent across both tools.

Sourced from Docs by LangChain (MCP)

I create automation rules on the production project that filter by conditions such as high latency, high token usage or errors, scoped by feature metadata, with a sampling rate so volume stays manageable. Matching runs go into an annotation queue, where reviewers use a rubric to judge whether the cost was justified, for example whether the long answer or extra tool turns were needed. Good cases are exported to a dataset. That dataset becomes the regression set for evaluate() experiments when prompts or models change. So production cost outliers directly shape the tests that guard future releases.

Sourced from Docs by LangChain (MCP)

Per feature. A short autocomplete and a multi-step research agent have very different acceptable latencies and token profiles. A single app-wide threshold would either alert constantly on the agent or never catch regressions in the fast feature. With feature metadata on every run, I chart and alert on p50 and p99 per feature, with thresholds based on each feature's own baseline and user expectations. For streamed features, the budget includes time to first token. Per-feature budgets also give each owning team clear accountability when its numbers move.

Sourced from Docs by LangChain (MCP)

1. ALANGSMITH_DEBUG
2. BLANGCHAIN_VERBOSE
3. COTEL_TRACES_ENABLED
4. DLANGSMITH_TRACINGCorrect answer

Why: Setting LANGSMITH_TRACING, together with an API key, turns on tracing to LangSmith.

Sourced from Docs by LangChain (MCP)

1. ATo cache responses and reduce cost
2. BTo automatically capture model calls, including token usage, as traced runsCorrect answer
3. CTo route requests to a cheaper model
4. DTo encrypt prompts before sending them

Why: wrap_openai traces each client call so token counts are recorded automatically, and those counts are used for cost.

Sourced from Docs by LangChain (MCP)

1. AA tag named 'cost'
2. Busage_metadata plus ls_provider and ls_model_nameCorrect answer
3. CThe full HTTP response headers
4. DOnly the request latency

Why: Token usage plus provider and model name let LangSmith match the call to the model pricing map.

Sourced from Docs by LangChain (MCP)

1. ARerun the request with caching enabled
2. BRaise the model's max tokens
3. COpen the trace tree and compare child-run latenciesCorrect answer
4. DCheck the project's total monthly cost

Why: The trace tree shows each nested run's latency, exposing slow retrieval, retries or extra agent turns.

Sourced from Docs by LangChain (MCP)

1. AIncrease the online evaluator sampling rate
2. BEnable a pairwise annotation queue
3. CSwitch the project to a new workspace
4. DReturn usage_metadata and set ls_provider and ls_model_name on the runCorrect answer

Why: LangSmith computes cost from recorded token usage and the model identity matched against the pricing map, so custom runs need usage_metadata plus ls_provider and ls_model_name.

Sourced from Docs by LangChain (MCP)

1. AIt keeps dev and load-test traffic from distorting production dashboards and alertsCorrect answer
2. BProjects are required for the @traceable decorator to work
3. CIt makes tracing faster
4. DLangSmith allows only one environment per API key

Why: Separate projects isolate production metrics, alerts and rules from non-production traffic.

Sourced from Docs by LangChain (MCP)

1. ADatasets
2. BAnnotation queues
3. CExperiments
4. DThreadsCorrect answer

Why: Threads group multi-turn conversations, allowing per-conversation cost and latency analysis.

Sourced from Docs by LangChain (MCP)

1. ANumber of child runs
2. BTotal tokens per trace
3. CError rate
4. DTime to first tokenCorrect answer

Why: Time to first token measures when output starts appearing, which is what users perceive as responsiveness.

Sourced from Docs by LangChain (MCP)

1. AGrouping by request ID
2. BNo grouping, just the overall total
3. CGrouping by day of week only
4. DGrouping by model and by the feature metadata keyCorrect answer

Why: Grouping by model and feature isolates whether one model or feature caused the change.

Sourced from Docs by LangChain (MCP)

1. Ap99 is cheaper to compute
2. Bp50 is not supported for LLM runs
3. Cp99 replaces the need for error rate
4. Dp99 reveals tail problems such as retries or long contexts that p50 hidesCorrect answer

Why: Long-tailed LLM latency means the slowest requests can regress while the median stays flat.

Sourced from Docs by LangChain (MCP)

1. AIncrease the sampling rate of online evaluators
2. BSkip alerting for that metric
3. CRun a scheduled job that queries runs via the SDK and posts to the same webhookCorrect answer
4. DAsk users to report slowness

Why: A small SDK-based job can compute custom metrics and deliver through the same alert channel.

Sourced from Docs by LangChain (MCP)

1. AThe pricing map updates automatically
2. BCost and latency get misattributed, so model decisions are based on wrong numbersCorrect answer
3. CNone, because cost is averaged across models
4. DLatency becomes more accurate

Why: Each attempt must record the model actually used, or the numbers teams act on are wrong.

Sourced from Docs by LangChain (MCP)

1. AAutomatically switch models when cost rises
2. BDelete old traces to save storage
3. CRewrite prompts to reduce tokens
4. DFilter runs, for example slow or expensive ones, with a sampling rate and add them to a dataset or annotation queueCorrect answer

Why: Automation rules select matching runs, sampled, and route them to datasets or annotation queues.

Sourced from Docs by LangChain (MCP)

1. AFrom the model name
2. BFrom the project name
3. CFrom the calling entry point, propagated to child runsCorrect answer
4. DHardcoded inside the helper

Why: Setting feature at the caller and propagating it ensures the helper's runs are attributed to the right feature.

Sourced from Docs by LangChain (MCP)

1. AThe project has too many runs
2. BThe run took too long
3. CThe model name doesn't match an entry in the pricing mapCorrect answer
4. DTracing is disabled

Why: Cost is computed by matching provider and model name to the pricing map, so a missing or mismatched name yields no cost.

Sourced from Docs by LangChain (MCP)

1. AA tracing outage or broken instrumentation could otherwise look like low costCorrect answer
2. BLow volume increases latency
3. CIt is required to enable alerts
4. DLow volume always means a cost spike

Why: Missing data can masquerade as healthy low numbers unless volume drops are detected.

Sourced from Docs by LangChain (MCP)

1. AAdd a second, separate tracing SDK to every service
2. BExport spans through an OTel collector that fans out to both the APM and the LangSmith OTLP endpointCorrect answer
3. CCopy APM logs into a LangSmith dataset nightly
4. DUse pytest integration to replay production traffic

Why: LangSmith accepts OTLP natively, and a collector can send identical spans to multiple backends without double instrumentation.

Sourced from Docs by LangChain (MCP)

1. ASampling is required for dashboards to render
2. BRules without sampling disable alerts
3. CUnsampled rules cannot read metadata
4. DMatched traces are upgraded to extended retention, which raises volume and costCorrect answer

Why: Running an online evaluator upgrades a trace to extended retention, so evaluating every run increases retained data and cost; filters and sampling keep it targeted.

Sourced from Docs by LangChain (MCP)

1. AAn annotation queue reviewer count
2. BA dataset and an evaluator
3. CA prompt tag and commit
4. DA threshold and a time window, plus a delivery target such as a webhookCorrect answer

Why: Alerts compare metrics against thresholds over a time window and deliver to a webhook or paging tool.

Sourced from Docs by LangChain (MCP)

1. AIt converts spans into datasets automatically
2. BBoth tools share trace IDs from a single instrumentation pathCorrect answer
3. CIt removes the need for an API key
4. DIt doubles the sampling rate

Why: Fan-out sends the same spans to both tools, so infrastructure and LLM views share trace IDs.

Sourced from Docs by LangChain (MCP)

1. ACaching is not supported by evaluate()
2. BCaching increases token usage
3. CCached responses don't reflect real model latency or token costCorrect answer
4. DCaching disables repetitions

Why: Cached outputs skip real model calls, so measured latency and cost would not reflect production behaviour.

Sourced from Docs by LangChain (MCP)

1. AIt is moved to a different project
2. BIts cost is set to zero
3. CIt is upgraded to extended retentionCorrect answer
4. DIt is deleted after evaluation

Why: Running an online evaluator upgrades a trace to extended retention, which affects storage and cost planning.

Sourced from Docs by LangChain (MCP)

1. ADisable the integration and use OTel only
2. BTrace only the final output
3. CAdd @traceable to every node in addition to the integration
4. DRely on the LangGraph integration and add @traceable only for custom code outside the frameworkCorrect answer

Why: The integration already traces nodes and model calls; extra decorators on the same steps create duplicate runs.

Sourced from Docs by LangChain (MCP)

1. Araw request IDCorrect answer
2. Btenant tier
3. Cfeature
4. Dmodel

Why: Request IDs are unbounded in cardinality, which makes grouped charts unreadable.

Sourced from Docs by LangChain (MCP)

1. AThey should fail fast
2. BThey should queue until LangSmith recovers
3. CThey should switch to a cheaper model
4. DThey should continue normally; only trace data is affectedCorrect answer

Why: Tracing runs in the background and must not sit on the request's critical path.

Sourced from Docs by LangChain (MCP)

1. AConversation history is resent, so input tokens grow each turnCorrect answer
2. BAnnotation queues add tokens
3. CThe pricing map changes during the conversation
4. DThreads add overhead per turn

Why: Resending growing history increases input tokens per turn, which threads make visible per conversation.

Sourced from Docs by LangChain (MCP)

1. AIn the LangSmith dashboard after ingestion
2. BIn the annotation queue
3. CIn the Prompt Hub
4. DIn the app or the OTel collector before exportCorrect answer

Why: Redacting before export ensures sensitive data never leaves your infrastructure.

Sourced from Docs by LangChain (MCP)

1. AGroup cost and latency by prompt version to spot prompt-driven regressionsCorrect answer
2. BAutomatically roll back prompts
3. CReduce token usage
4. DSkip evaluate() experiments

Why: With prompt version on runs, dashboards can show whether a new prompt commit changed tokens or latency.

Sourced from Docs by LangChain (MCP)

1. AEvaluators only work on sampled data
2. BSampling improves evaluator accuracy
3. CTo control judge cost and extended-retention volume while focusing on runs that matterCorrect answer
4. DFilters are required for dashboards

Why: Evaluators add model cost and upgrade traces to extended retention, so targeted sampling keeps them affordable.

Sourced from Docs by LangChain (MCP)

1. AError rate grouped by model and featureCorrect answer
2. BPrompt version count
3. CDataset size
4. DNumber of annotation reviewers

Why: Error rate by model and feature shows when cost or latency spikes line up with provider failures and retries.

Sourced from Docs by LangChain (MCP)

1. A@pytest.mark.trace
2. B@pytest.mark.langsmithCorrect answer
3. C@pytest.mark.llm
4. D@pytest.mark.cost

Why: @pytest.mark.langsmith turns test files into LangSmith datasets and experiments.

Sourced from Docs by LangChain (MCP)

1. AAdd them to the annotation queue
2. BSend indexing to a separate project or give it a distinct feature valueCorrect answer
3. CDon't trace embeddings
4. DLog embeddings under the chat model's name

Why: Separating indexing keeps batch spikes from looking like user-facing cost regressions.

Sourced from Docs by LangChain (MCP)

1. AExactly twoCorrect answer
2. BExactly three
3. COne, against a reference
4. DAny number

Why: Pairwise queues compare exactly two experiments.

Sourced from Docs by LangChain (MCP)