Prompt versioning in the Prompt Hub

I'd decouple prompt content from code. Prompts live in the Prompt Hub, every change is an immutable commit, and the app pulls by tag, for example name:prod, caching it with a short TTL. Promotion means moving the staging tag to a commit, testing there, then moving prod to that same commit, but only after an evaluate() experiment on a regression dataset beats the current prod baseline. Every trace records the commit hash in metadata, and online evaluators sample production runs per commit with alerts. If scores drop, the owner moves the prod tag back to the previous known-good commit and apps pick it up on the next refresh. A last-good cache protects against Hub outages. Tag moves are permissioned and audited like deploys.

Sourced from Docs by LangChain (MCP)

The engineer edits messages, variables and model settings in the Playground and sanity-checks the output against a few saved inputs. They commit to the Prompt Hub with a clear message, which creates an immutable commit identified by a hash. CI, triggered manually or by a prompt-commit webhook, pulls that exact hash and runs evaluate() on a regression dataset with code and LLM-as-judge evaluators. The resulting experiment is compared with the current prod commit's experiment. If key scores hold and no critical example regresses, the staging tag moves first, then the prod tag. Apps pull name:prod on a TTL and cache it. Traces record the commit hash, and online evaluators watch for drops.

Sourced from Docs by LangChain (MCP)

Production only resolves the prod tag, so the guarantee depends on controlling who can move that tag and when. I restrict tag moves on prod to an owner group or a CI service identity. That identity only moves the tag after it has found a passing experiment for that exact commit hash. I would also keep an allowlist of evaluated commit hashes. The app or a watcher checks each resolved hash against that list and alerts, or refuses, if prod points at something unknown. Commits never move tags by themselves. Edits in the Playground create new commits but never change what prod serves.

Sourced from Docs by LangChain (MCP)

Immutability makes a commit hash a reliable identifier. The evaluation result for a hash, the traces that cite it, and a later rollback target all refer to exactly the same content. If commits could be edited in place, a passing experiment would no longer prove anything about what is serving traffic. A trace's metadata could also point at text that has since changed. Because old commits stay retrievable, rollback is just pointing a tag at an earlier hash. You never reconstruct an old prompt from memory or from git history, and audits can reproduce any production output's prompt exactly.

Sourced from Docs by LangChain (MCP)

The TTL sets the trade-off between how fast rollbacks take effect and how much load and dependency you put on the Hub. A short TTL means a moved prod tag reaches the fleet quickly, which matters most during an incident. However, every instance pulls more often. A long TTL reduces calls and Hub dependency but stretches the window in which bad prompts keep serving after rollback. I pick the TTL from the recovery-time target the team agrees on and keep it short. I serve from cache between refreshes, refresh asynchronously so requests never block on the Hub, and offer an explicit invalidation path for emergencies.

Sourced from Docs by LangChain (MCP)

At pull time, I resolve the tag to a concrete commit hash and keep the name, tag and hash together in the cached prompt object. Every traced call then attaches these as run metadata. You can do this through the @traceable decorator's metadata, the LangChain run config, or OpenTelemetry span attributes when exporting over OTLP. The hash must come from the object actually used for that request, not from a global variable read separately. Otherwise a cache refresh mid-request could mislabel a trace. I also record whether the prompt came from a fresh pull, the cache, or the fallback snapshot.

Sourced from Docs by LangChain (MCP)

First, I identify the bad commit from trace metadata and online evaluator alerts. Then I pick the previous known-good commit, usually the one prod pointed at before the promotion, which the promotion record should capture. An owner moves the prod tag back to that hash. No code changes and no deploy are needed. Instances pick it up on their next pull or cache refresh, so recovery time is bounded by the TTL plus any refresh delay. I confirm by watching new traces report the old hash and scores recover. Afterwards, I add the failing cases to the regression dataset.

Sourced from Docs by LangChain (MCP)

Tags are pointers that move; commit hashes are immutable. I use tags where I want a controlled, central switch, mainly production and staging services, because rollback becomes a tag move. I pin hashes wherever reproducibility matters more than flexibility: evaluation experiments, regression tests in CI, backtesting and incident investigations, so rerunning tomorrow tests exactly the same text and settings. I'd also pin a hash for a high-risk integration that must only change through a code review, or during a freeze. The key is that even tag-based production traffic logs the resolved commit hash in trace metadata, so I always know which immutable version served each request and can replay it precisely.

Sourced from Docs by LangChain (MCP)

I treat it as a regression test plus a staged rollout. First, run the candidate commit through evaluate() on a curated dataset that mixes golden cases and real production failures, using code evaluators for format and hard rules and LLM-as-judge for quality, with repetitions to smooth non-determinism. I compare that experiment against the current prod commit's experiment, looking at aggregate scores and per-example regressions, not only the average. Critical cases are blocking. If it passes, I move the staging tag and watch staging traces. Then I promote prod and monitor online evaluators filtered to the new commit for a defined window, with an alert threshold that triggers rolling the prod tag back. Hard numbers come from my own baseline, not guesses.

Sourced from Docs by LangChain (MCP)

A prompt is a named entity in the Prompt Hub. Each saved change creates a commit, which is an immutable snapshot of the messages, variables and model settings, identified by a hash and ordered in a history. Tags are mutable, named pointers such as staging and prod, and each tag references one commit at a time. Applications refer to name:tag for indirection or name:hash for pinning. Promotion and rollback are therefore pointer updates, not content changes. Experiments and production traces should both reference the commit hash, never just the tag, because a tag's meaning changes over time.

Sourced from Docs by LangChain (MCP)

This is the dangerous case, because the 'last good cached version' does not exist yet. I would ship a fallback snapshot with the deployment. It is a known-good commit, pinned by hash, that was itself evaluated, and the artifact records its hash. On cold start, the app tries the Hub first and falls back to the snapshot if the Hub cannot be reached. Traces then mark the source as fallback along with the hash. If even that is not acceptable, the service should fail closed and report unhealthy rather than serve an unknown prompt. It must never fall back to an untagged or latest commit.

Sourced from Docs by LangChain (MCP)

I combine three sources: curated cases that encode product requirements, examples taken from production traces, and synthetic data to fill gaps. The production traces should include failures found through annotation queues or user feedback. Each example has inputs and, where possible, reference outputs. I tag a subset as critical, such as safety, compliance or high-value flows, because a regression there blocks promotion regardless of averages. The dataset is versioned and grows over time. Every incident should add an example so the same failure cannot ship twice. I also keep it small enough that CI evaluation finishes in a time developers will accept.

Sourced from Docs by LangChain (MCP)

I run evaluate() with repetitions so each example is scored several times. That way I compare distributions instead of single noisy samples. Both experiments should use the same dataset version and evaluator versions, and ideally the same model settings except where the change is intentional. Otherwise, the comparison mixes several changes together. Caching helps keep reruns cheap and consistent. I define promotion thresholds that allow for normal variance, but I treat critical examples strictly. If a critical example fails in any repetition, I investigate rather than averaging the failure away.

Sourced from Docs by LangChain (MCP)

I treat them as useful but fallible. I use prebuilt templates as a starting point, add few-shot examples that reflect our own grading standards, and calibrate them against human labels from annotation queues. Code evaluators come first for anything deterministic, such as format, JSON schema, required fields or banned content, because they are cheap and exact. I version the judge prompt itself in the Prompt Hub. That way a judge change is visible and does not silently shift scores. When the judge and humans disagree, I fix the judge before trusting it to gate releases.

Sourced from Docs by LangChain (MCP)

Offline evaluation tests the prompt against a dataset. Staging tests it inside the real application. Staging catches integration problems a dataset may miss, such as a renamed template variable that the code does not supply, a model setting that breaks tool calling, a longer output that breaks a parser, or latency and cost changes. Because staging and prod both resolve tags the same way, staging proves the pull, cache and tracing paths work for that commit. Then prod is moved to the same hash, not a re-commit. What was exercised is exactly what ships.

Sourced from Docs by LangChain (MCP)

With TTL-based pulls, the fleet converges gradually, so for up to one TTL both the old and new commits are live. That is usually acceptable because both are evaluated versions. Every trace records its commit hash, so analysis can split traffic cleanly. It becomes a problem for multi-turn conversations or multi-step chains, where switching mid-flow could produce inconsistent behaviour. There, I resolve the version once at the start of a thread or request and pin that hash for its duration. Dashboards filtered by commit let us see the transition rather than treating it as noise.

Sourced from Docs by LangChain (MCP)

I attach online evaluators to the production project with a sampling rate rather than scoring everything. I use filters on the prompt commit metadata so the new commit can be compared directly with the previous one. Cheap code evaluators can run at a higher rate, and LLM-as-judge evaluators at a lower one. After a promotion, I might temporarily raise sampling for the new commit to get signal faster. Dashboards show scores by commit, and alerts fire on drops. Note that running an online evaluator upgrades a trace to extended retention, so sampling also controls storage cost.

Sourced from Docs by LangChain (MCP)

I separate three capabilities: editing and committing, moving the staging tag, and moving the prod tag. Many engineers can commit, because commits are harmless until tagged. Moving prod should be limited to owners or a CI identity that enforces the evaluation gate. Applications get read-only API keys scoped to pulling prompts and writing traces. Keys live in a secrets manager and are rotated. I keep an audit trail of tag moves and alert on unexpected changes to prod. Prompts themselves must never contain secrets, because they are visible to anyone with read access and appear in traces.

Sourced from Docs by LangChain (MCP)

Traces can contain user PII, and copying them into datasets spreads that data into a longer-lived, more widely shared place. I redact or mask sensitive fields before tracing where possible, or at least before export to datasets or annotation queues. I limit who can view the production project and its queues. For India, I would design with the DPDP Act's consent and purpose-limitation principles in mind, and keep datasets to the minimum needed to reproduce a behaviour. Retention also matters. Online evaluators move traces to extended retention, so scoring sensitive traffic means keeping it longer.

Sourced from Docs by LangChain (MCP)

The prompt's input variables are a contract with the code, so a rename is effectively a breaking API change. The regression dataset should use the same input keys the application sends. Then evaluate() fails on a missing variable instead of passing silently. Staging catches anything the dataset missed. For an intentional rename, I make the code accept both names first, deploy that, and only then promote the prompt. That way the tag move and the deploy are not coupled. Rolling back the prompt must also stay compatible with the current code, which is another reason to keep both names supported for a while.

Sourced from Docs by LangChain (MCP)

Averages can hide serious failures. A prompt can raise the mean score while breaking a refusal case, a compliance disclaimer, or one high-value customer flow. So I mark certain examples as critical and require each to pass, or at least not score worse than the prod commit's experiment. In the experiment comparison, I look at per-example diffs, not only aggregate metrics. Any critical example that goes from pass to fail blocks promotion until someone reviews it. Non-critical regressions are acceptable if they are understood and outweighed by improvements, and the reason is recorded in the promotion decision.

Sourced from Docs by LangChain (MCP)

Keeping them in the commit means the evaluated unit is the whole behaviour, meaning text plus model plus parameters. The hash then fully describes what produced an output, and rollback restores both together. If the model is chosen in code instead, a code deploy can change behaviour under an unchanged prompt hash. The trace would then not explain the output, and evaluation results would not transfer. Some teams keep the model in code for cost or provider control. If so, I record the model in trace metadata too, and rerun the prompt's experiment whenever the model changes.

Sourced from Docs by LangChain (MCP)

The webhook tells CI that a commit exists. CI should then pull that specific commit by hash, never 'latest', because another commit could land between the event and the pull. The job should be idempotent per hash, so retries or duplicate deliveries do not produce conflicting results. It runs evaluate() against the regression dataset, stores the experiment linked to the hash, and reports pass or fail. The webhook endpoint should verify the sender and must never move tags automatically. Tag moves for prod should stay a separate, authorised step that checks the experiment result for that exact hash.

Sourced from Docs by LangChain (MCP)

Pull volume is roughly the number of instances times the number of prompts each uses, divided by the TTL, plus a burst at deploy time when everything starts together. Request traffic does not matter as long as requests are served from cache. To reduce load, I lengthen the TTL within the rollback target and add jitter so instances do not refresh in sync. I refresh asynchronously in the background and avoid pulling per request. For large fleets, a shared internal cache or sidecar can pull once and serve many instances, at the cost of an extra layer to keep consistent.

Sourced from Docs by LangChain (MCP)

I would add a canary tag pointing at the candidate commit. The application routes a small, defined share of requests to name:canary and the rest to name:prod. The routing decision lives in code or feature-flag config, while the content stays in the Hub. Both paths record their commit hash, so online evaluators filtered by commit give a side-by-side comparison on real traffic. For qualitative judgement, pairwise annotation queues let reviewers compare outputs from two experiments. The canary commit must already have passed offline evaluation. Canarying limits the blast radius but does not replace the gate.

Sourced from Docs by LangChain (MCP)

Generally, no. Changing instructions mid-conversation can cause inconsistent tone, forgotten constraints or contradictory behaviour, and it makes the thread hard to debug. I resolve the commit hash when the thread starts, store it with the conversation state, and pull or cache that hash for later turns. New threads get the new prod commit. The exception is a rollback for a harmful prompt. Then active threads should also switch, so I give the system a way to override pinned hashes. Thread-level online evaluation, filtered by commit, shows whether the new version behaves well across whole conversations.

Sourced from Docs by LangChain (MCP)

Backtesting replays real historical inputs through the candidate prompt and compares the results with what the current prod commit produced. I build a dataset from recent production traces, ideally including ones flagged by online evaluators or users. Then I run evaluate() on the candidate commit pulled by hash. Because traces carry the commit hash, I know exactly which version generated the historical outputs. This catches problems that curated cases miss, such as real input distributions, odd formatting and long tails. I use it alongside the regression dataset, not instead of it, and redact PII before converting traces to examples.

Sourced from Docs by LangChain (MCP)

They let the team write prompt checks as ordinary tests. With @pytest.mark.langsmith, or the Vitest and Jest equivalents, the test file's cases become a dataset and each run becomes an experiment in LangSmith. In CI, the test suite pulls the candidate commit by hash, runs the cases, and asserts on code-level properties. The results are logged as an experiment that can be compared with prod's. This suits deterministic checks and developer workflows. Broader quality judgements still rely on evaluate() with LLM-as-judge evaluators over the full regression dataset.

Sourced from Docs by LangChain (MCP)

Roll back first, fix second. Moving prod to the previous known-good commit stops the damage within one TTL, needs no deploy, and requires no new evaluation, because that commit was already evaluated. The hotfix then goes through the normal path: commit, evaluate against the regression dataset plus the new failing cases, staging, then prod. If there is no safe earlier version, for example because the bug has existed for a long time, an expedited evaluation on critical examples is the minimum. Skipping evaluation entirely breaks the core rule that prod never serves an unevaluated version.

Sourced from Docs by LangChain (MCP)

A comparison is only valid if both experiments ran on the same dataset version with the same evaluators. When the dataset grows, the prod commit's old experiment becomes a weaker baseline. So the gate should rerun the current prod commit, pulled by hash, on the current dataset alongside the candidate. Caching makes that cheaper. I also record the dataset version and evaluator versions with each experiment. Removing examples should require review, because quietly deleting a hard case is an easy way to make a weak prompt look good.

Sourced from Docs by LangChain (MCP)

LangSmith accepts OpenTelemetry traces through its OTLP endpoint, and a collector can fan out the same spans to LangSmith and other backends. I add the prompt name, tag and resolved commit hash as span attributes on the LLM call span, using values from the prompt object actually used. Then I check that these attributes show up as filterable metadata in the LangSmith project. That matters because online evaluator filters and dashboards depend on them. The rule is the same as with @traceable: one source of truth for the hash, attached at call time.

Sourced from Docs by LangChain (MCP)

I start from the trace in the LangSmith project. Its metadata gives the prompt name and exact commit hash, and the run tree shows inputs, retrieved context, tool calls and the model response. I open that commit in the Prompt Hub to see the exact messages and settings, and compare it with the previous commit to see what changed. If the commit looks fine, the cause may be the input, retrieval, or the model itself. I then add the case to the regression dataset, or send it to an annotation queue for labelling, so it guards future commits.

Sourced from Docs by LangChain (MCP)

I keep the same model: commits, staging and prod tags, and pull by tag. I add conventions and automation around it. Each prompt has a named owner, a regression dataset, and CI wiring through prompt-commit webhooks, so evaluation runs on every commit without manual steps. Promotion runs through one shared CI service that enforces the gate the same way for every prompt. Services pull only the prompts they need and cache them independently. Dashboards group online evaluation by prompt and commit. When prompts depend on each other, such as a router and its sub-prompts, I evaluate them together as a chain and promote them as a coordinated set.

Sourced from Docs by LangChain (MCP)

1. Aclient.pull_prompt("my-prompt:prod") while the prod tag is moved to an earlier commitCorrect answer
2. BHard-coding the prompt string copied from the Playground
3. Cclient.push_prompt("my-prompt") on every request
4. Dclient.pull_prompt("my-prompt") with no tag

Why: Pulling by tag resolves to whichever commit the tag points at, so moving the prod tag back changes what the app gets on its next pull without a code change.

Sourced from Docs by LangChain (MCP)

1. AA draft that disappears once the Playground session ends
2. BAn automatic move of the prod tag to the new content
3. CAn immutable version identified by a commit hashCorrect answer
4. DAn edit that overwrites the current prod version in place

Why: Each commit is immutable and has a hash, so earlier versions stay retrievable. Tags only move when someone moves them explicitly.

Sourced from Docs by LangChain (MCP)

1. ABy moving the prod tag to the evaluated commitCorrect answer
2. BBy deleting older commits so only the new one remains
3. CBy editing the existing prod commit directly
4. DBy redeploying the application with the new prompt text

Why: Promotion is a tag move. Staging is moved first, then prod is pointed at the same commit.

Sourced from Docs by LangChain (MCP)

1. ARecording the commit hash in trace metadata
2. BMoving staging before prod
3. CHaving the application pull the latest commit instead of the prod tagCorrect answer
4. DCaching the prod prompt for a short TTL

Why: The latest commit may not have been evaluated. Only the tag gated by evaluation should be served.

Sourced from Docs by LangChain (MCP)

1. AIt reduces token usage
2. BIt is required for the Playground to run
3. CIt ties every output to an exact immutable prompt version for debugging and per-version evaluationCorrect answer
4. DIt automatically moves the prod tag

Why: Tags move over time, so only the commit hash tells you exactly which prompt version produced a given run, enabling filtering, online evaluation per version and replay.

Sourced from Docs by LangChain (MCP)

1. AIt guarantees the evaluated content is exactly the commit that may be promotedCorrect answer
2. BHashes make LLM outputs deterministic
3. CPulling by hash is the only way to use the Playground
4. DTags cannot be read by the SDK

Why: A hash refers to one immutable version. A tag or latest pointer could change between the commit event and the evaluation.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai()
2. Btraceable()
3. Cevaluate()Correct answer
4. Dpull_prompt()

Why: evaluate() runs experiments over datasets with evaluators and supports repetitions, concurrency and caching.

Sourced from Docs by LangChain (MCP)

1. ABoth tags at the same time to avoid drift
2. BProd only, because staging is optional
3. CStaging first, exercised in the staging environment, then prod to the same commitCorrect answer
4. DProd first, then staging to match

Why: The workflow points staging at the new commit first, exercises it, and then moves prod to that same commit.

Sourced from Docs by LangChain (MCP)

1. AOn the next pull or cache refreshCorrect answer
2. BImmediately, because the Hub pushes the change into running processes
3. COnly after all instances are restarted manually
4. DOnly after the next code deploy

Why: Apps pull name:prod at startup or on a short TTL, so the change takes effect on the next pull or refresh.

Sourced from Docs by LangChain (MCP)

1. AThe size of the regression dataset
2. BThe number of evaluators configured
3. CThe prompt cache TTL and refresh behaviourCorrect answer
4. DThe length of the commit message

Why: Instances only see the moved tag when they next pull, so the TTL sets the propagation window.

Sourced from Docs by LangChain (MCP)

1. ABlock promotion because a critical example regressedCorrect answer
2. BPromote, because the average improved
3. CPromote to prod but not to staging
4. DDelete the critical example from the dataset

Why: Promotion requires key scores to hold or improve and no critical example to regress.

Sourced from Docs by LangChain (MCP)

1. AOnly in the application's startup log
2. BIn the prompt's commit message
3. CIn the regression dataset's reference outputs
4. DIn the trace's metadata in the LangSmith projectCorrect answer

Why: Recording it in trace metadata ties each output to an exact version and allows filtering by commit.

Sourced from Docs by LangChain (MCP)

1. AFall back to a prompt hard-coded by a developer at runtime
2. BKeep serving the last good cached versionCorrect answer
3. CStop serving all requests until the Hub returns
4. DPull the latest commit from another region

Why: The workflow says to fall back to the last good cached version when the Hub is unreachable.

Sourced from Docs by LangChain (MCP)

1. ALet anyone with Playground access move the prod tag
2. BStore the Hub API key in the prompt text
3. CAllow broad commit access but restrict prod tag moves to owners or a gated CI identityCorrect answer
4. DGive every app API key permission to move tags

Why: Commits are harmless until tagged, so control belongs on who can move prod.

Sourced from Docs by LangChain (MCP)

1. ARedeploy the application with the prompt hard-coded
2. BDisable tracing to reduce load
3. CPush an unevaluated hotfix commit and move prod to it
4. DMove the prod tag back to the previous known-good commitCorrect answer

Why: Rolling back to an already-evaluated commit is fast, needs no deploy, and keeps the no-unevaluated-version rule.

Sourced from Docs by LangChain (MCP)

1. APull the prompt on every request with retries
2. BCache the pulled prompt with a TTL and fall back to the last good versionCorrect answer
3. CDisable tracing
4. DAlways use the latest commit instead of a tag

Why: A local cache with a last-known-good fallback removes the Hub from the hot path, so outages or latency do not break requests.

Sourced from Docs by LangChain (MCP)

1. AHuman reviewer in a single-run queue
2. BPairwise annotation queue
3. CCode rule evaluatorCorrect answer
4. DLLM-as-judge with few-shot examples

Why: Deterministic structural checks are cheap and exact when written as code evaluators.

Sourced from Docs by LangChain (MCP)

1. ATo create additional commits in the Hub
2. BTo reduce the effect of non-deterministic model outputs on the comparisonCorrect answer
3. CTo increase the online evaluator sampling rate
4. DTo bypass the caching layer

Why: Repetitions score each example several times, so comparisons are less sensitive to random variation.

Sourced from Docs by LangChain (MCP)

1. AA thread grouping event
2. BAn OTLP collector fan-out
3. CA prompt-commit webhookCorrect answer
4. DAn annotation queue reservation

Why: The workflow allows a prompt-commit webhook into CI to start the evaluation.

Sourced from Docs by LangChain (MCP)

1. ATo let the evaluator move the prod tag automatically
2. BFilters are required for tracing to work
3. CSo score drops can be attributed to the new commit versus the previous oneCorrect answer
4. DTo stop traces from reaching extended retention

Why: Filtering by commit separates traffic by version, so a regression can be linked to the promotion.

Sourced from Docs by LangChain (MCP)

1. AIt deletes the trace after scoring
2. BIt moves the trace to a different project
3. CIt upgrades the trace to extended retentionCorrect answer
4. DIt converts the trace into a dataset example automatically

Why: Running an online evaluator upgrades a trace to extended retention, which affects storage cost.

Sourced from Docs by LangChain (MCP)

1. AUnsampled runs cannot be traced
2. BIt controls evaluation cost and retention while still giving a signalCorrect answer
3. CSampling is needed to record commit hashes
4. DSampled runs are excluded from dashboards

Why: Scoring a sampled share keeps judge cost and extended-retention storage manageable while still detecting drops.

Sourced from Docs by LangChain (MCP)

1. Awrap_openai
2. Bagentevals trajectory evaluatorsCorrect answer
3. CPairwise annotation queues
4. DVitest integration

Why: agentevals provides trajectory evaluators, such as trajectory LLM-as-judge, for agent paths.

Sourced from Docs by LangChain (MCP)

1. AIt moves the staging tag when tests pass
2. BIt disables tracing during tests
3. CIt pulls the prod prompt automatically
4. DIt turns the test file's cases into a dataset and its runs into experimentsCorrect answer

Why: The pytest integration logs test cases as a dataset and runs as experiments in LangSmith.

Sourced from Docs by LangChain (MCP)

1. ALANGSMITH_PROMPT_TAG
2. BLANGSMITH_EVAL_MODE
3. CLANGSMITH_COMMIT_HASH
4. DLANGSMITH_TRACINGCorrect answer

Why: Tracing is enabled with LANGSMITH_TRACING, together with decorators such as @traceable or wrappers.

Sourced from Docs by LangChain (MCP)

1. APin the commit hash resolved at thread start for the rest of that threadCorrect answer
2. BAlternate between old and new commits on each turn
3. CSwitch every active thread to the new commit on its next turn
4. DRestart all conversations when prod moves

Why: Pinning per thread avoids inconsistent behaviour, while new threads pick up the new prod commit.

Sourced from Docs by LangChain (MCP)

1. AThe prod tag stops working after a dataset change
2. BRe-evaluation moves the staging tag back
3. CComparisons are only valid when both experiments ran on the same dataset versionCorrect answer
4. DOld experiments are deleted when datasets change

Why: A baseline scored on an older dataset is not a like-for-like comparison with the candidate.

Sourced from Docs by LangChain (MCP)

1. AOffline evaluation using the app's input keys, followed by stagingCorrect answer
2. BWriting a longer commit message
3. CRaising the cache TTL
4. DOnline evaluators after promotion

Why: Offline evaluation fails on missing variables if the dataset uses the app's real keys, and staging tests real integration.

Sourced from Docs by LangChain (MCP)

1. ABy importing log files into a dataset
2. BOnly through the LangChain integration
3. CThrough a prompt-commit webhook
4. DThrough an OTLP endpoint, optionally fanned out via a collectorCorrect answer

Why: LangSmith supports native OpenTelemetry through an OTLP endpoint, and a collector can fan out traces to several backends.

Sourced from Docs by LangChain (MCP)

1. APull the prompt on every request
2. BRemove caching and rely on the Hub's availability
3. CPull by latest instead of by tag
4. DAdd jitter to cache refreshes so instances do not all pull at onceCorrect answer

Why: Jittered, cached refreshes spread pulls over time instead of creating synchronised bursts.

Sourced from Docs by LangChain (MCP)

1. AExactly two experimentsCorrect answer
2. BOne experiment against production traces
3. CAny number of experiments
4. DAt least three experiments

Why: Pairwise queues compare exactly two experiments.

Sourced from Docs by LangChain (MCP)

1. AThreadsCorrect answer
2. BDatasets
3. CTags
4. DProjects

Why: Threads group multi-turn conversations, while projects group runs and traces more broadly.

Sourced from Docs by LangChain (MCP)

1. AOnly manually written examples
2. BOnly synthetic data
3. COnly experiment outputs
4. DCurated cases, production traces or synthetic dataCorrect answer

Why: Datasets can be built from curated cases, production traces or synthetic data.

Sourced from Docs by LangChain (MCP)