LangSmith blueprint · Software / AI applications
Evaluating a RAG application
Runs a versioned dataset of questions with reference answers through several RAG configurations using LangSmith evaluate(), then scores each run for correctness, groundedness and retrieval relevance with LLM-as-judge evaluators. It must never declare a winning chunk size or retriever from scores on different dataset versions, single noisy runs or a judge nobody has checked against human labels.
- Steps
- 9
- Subjective questions
- 33
- Objective questions
- 33
The problem and the outcome
Problem: Teams change chunk sizes, embedding models and retrievers based on a few manual spot checks. Regressions in answer quality or hallucinations then reach users unnoticed.
Outcome: Every retrieval or prompt change is tested as a named experiment against the same dataset and compared side by side on three separate quality signals. Winners are promoted on evidence, and CI blocks regressions before merge.
Architecture
How it works, step by step
- Build a LangSmith dataset of examples with a question as input and a reference answer as output, drawn from curated cases, production traces exported through an annotation queue, and reviewed synthetic questions. Tag the dataset version used as the benchmark.
- Define the configurations to compare, such as chunk size and overlap crossed with retriever type (dense, keyword, hybrid, with or without a reranker). Build a separate index per configuration so variants never share stale chunks.
- Wrap the RAG pipeline as a target function decorated with @traceable that takes inputs and returns both the answer and the retrieved documents. Evaluators need the documents as well as the answer.
- Call evaluate() once per configuration with the target, the pinned dataset, the evaluators, an experiment_prefix and metadata naming the chunk size and retriever. Use num_repetitions to measure run-to-run noise and max_concurrency to control rate limits.
- Score each run with three LLM-as-judge evaluators. Correctness compares the answer with the reference. Groundedness checks that each claim is supported by the retrieved documents. Retrieval relevance checks the documents against the question. Each judge returns a structured score and its reasoning.
- Open the experiments in the comparison view. Review the aggregate scores, latency and token cost from the traces, then drill into examples where configurations disagree.
- Treat the current production configuration as the baseline experiment. Only promote a variant that improves the target signal without regressing groundedness, and record the decision.
- Send close or surprising results to a pairwise annotation queue containing exactly the two experiments. Human preferences confirm the winner and help calibrate the judges.
- Turn key examples into pytest tests with @pytest.mark.langsmith so every pull request runs as an experiment. After release, add online evaluators with a sampling rate on production runs to catch drift.
Tech stack
- LangSmith datasets, experiments and comparison view
- langsmith SDK evaluate() and @traceable
- LLM-as-judge evaluators with structured output
- LangChain or LangGraph retrieval pipeline
- Vector store plus keyword or hybrid retriever
- LangSmith annotation queues (single-run and pairwise)
- pytest with @pytest.mark.langsmith in CI
- Prompt Hub for versioned judge prompts
Key design decisions
- Use three separate evaluators instead of one overall quality score, so a retrieval failure and a generation failure can be told apart.
- Have judges return binary or small-scale scores with reasoning instead of a 1 to 10 score, because coarse scales are easier to calibrate against humans.
- Store judge prompts in the Prompt Hub with commits and tags, so a change to a judge is versioned and never silently shifts scores.
- Pin a dataset version for each benchmark, and compare experiments only when they ran on the same version.
- Use num_repetitions on candidate configurations and treat differences within run-to-run noise as ties.
Common mistakes
- Using the same model as both generator and judge, which can reward its own style. Validate judges on a human-labelled sample and track their agreement.
- Groundedness scored without the retrieved documents, or correctness scored without a reference, which turns both into guesses about plausibility.
- Reference answers that go stale when the source documents change, so correct new answers get marked wrong.
- Datasets built only from easy synthetic questions, which miss multi-hop, ambiguous and unanswerable cases where retrievers differ most.
- Comparing chunk sizes while also changing the prompt or model, which makes the result impossible to attribute.
What to measure
- Correctness pass rate per experiment against the pinned dataset version
- Groundedness rate (share of answers fully supported by the retrieved documents)
- Retrieval relevance rate per retriever and chunk-size configuration
- Judge agreement with human labels from annotation queues
- p95 latency and token cost per example, taken from experiment traces
Most-asked interview questions
Subjective questions 33
1How would you evaluate a RAG system, and why split it into separate metrics?Asked almost every time
I separate retrieval quality from generation quality, because they fail differently and have different fixes. I build a dataset of questions with reference answers, then wrap the pipeline as a traced target that returns both the answer and the retrieved documents. Three LLM-as-judge evaluators score each run. Retrieval relevance asks whether the documents fit the question. Groundedness asks whether the answer is supported by those documents. Correctness asks whether the answer matches the reference. If relevance is low, I tune chunking or the retriever. If relevance is fine but groundedness is low, the model is hallucinating and I adjust the prompt or model. A single blended score would hide which of these is happening.
Sourced from Docs by LangChain (MCP)
2Why build a separate index for each chunking and retriever configuration instead of reusing one index?Asked almost every time
Chunk size and overlap change the actual units stored in the index, so a shared index would mix chunks from different settings or leave stale chunks from an earlier build. Results would then reflect contamination, not the configuration you meant to test. A separate index per configuration means each experiment retrieves only from chunks built with its own settings and embedding model. It also makes runs reproducible: I record the index identifier, chunk size, overlap and retriever type in the experiment metadata. The cost is extra embedding work and storage, but that is cheap compared with promoting a winner based on an index that never matched production.
Sourced from Docs by LangChain (MCP)
3Why must the target function return the retrieved documents as well as the answer?Asked almost every time
Two of the three evaluators cannot work without them. Groundedness checks whether every claim in the answer is supported by the retrieved documents, and retrieval relevance judges those documents against the question. If the target returned only the answer, I could score correctness against the reference but could not tell whether a wrong answer came from poor retrieval or poor generation. Returning the documents in the output dictionary makes them available to evaluators directly, and the @traceable decorator also records the retrieval step in the trace, so when I drill into a failing example I can see exactly what the model was given.
Sourced from Docs by LangChain (MCP)
4How would you build the evaluation dataset for a RAG application?Asked almost every time
I combine three sources. Curated cases cover known important and tricky questions that domain experts write with reference answers. Production traces give realistic phrasing; I route candidates through an annotation queue so a reviewer checks them, writes or corrects the reference, and exports them to the dataset. Synthetic questions generated from the corpus fill coverage gaps, but each is reviewed before it enters, because generated questions can be trivial or have wrong references. Each example has the question as input and the reference answer as output, plus metadata such as source and topic so I can slice results. Finally I tag the dataset version used as the benchmark so every experiment runs on the same examples.
Sourced from Docs by LangChain (MCP)
5What rule do you use to decide whether a new RAG configuration gets promoted?Asked almost every time
The current production configuration is the baseline experiment, run on the same tagged dataset version with the same evaluators. A variant is promoted only if it improves the target signal, say correctness or retrieval relevance, by more than the run-to-run noise I measured with repetitions, and does not regress groundedness. I also check latency and token cost from the traces, because a small quality gain may not justify a slower reranker. Close or surprising results go to a pairwise annotation queue for human confirmation. Finally I record the decision: experiment names, dataset version, scores, noise, and the reasoning, so anyone can audit or reverse it later.
Sourced from Docs by LangChain (MCP)
6How do you know your LLM-as-judge evaluators are trustworthy?Very often asked
I treat the judge as a model that needs its own evaluation. I label a sample of runs by hand, usually through an annotation queue with a clear rubric, and measure how often the judge agrees with the human labels. Where they disagree, I read the judge's reasoning, tighten the rubric, add few-shot examples drawn from the hard cases, and commit the new prompt version in the Prompt Hub. I prefer binary or small-scale outputs because they calibrate more reliably. Where possible, I use a different model family from the generator to reduce self-preference. I re-check agreement whenever I change the judge prompt or model.
Sourced from Docs by LangChain (MCP)
7Walk me through comparing chunk sizes and retrievers without fooling yourself.Very often asked
I change one factor at a time and keep everything else fixed: the same dataset version, prompt, generator model and judge prompts. Each configuration gets its own freshly built index and its own experiment, with metadata recording the chunk size and retriever. I run repetitions to see the noise, and I treat differences inside that noise as ties. In the comparison view, I look at all three scores together with latency and cost, because a bigger chunk might raise relevance but slow the system or add distracting context. Then I read the examples where configurations disagree. For close calls, a pairwise annotation queue lets humans pick between the two experiments before I promote anything.
Sourced from Docs by LangChain (MCP)
8How do you estimate the number of model calls an evaluation sweep will make?Very often asked
I multiply it out before running. Target calls are examples times configurations times repetitions. Each target call then triggers one call per LLM-as-judge evaluator, here three, so judge calls are that product times three. Retrieval and embedding calls add to it, plus a one-off cost to build each index. I do not guess the absolute numbers; I plug in the real dataset size and grid. That estimate drives decisions: I may prune the configuration grid, run fewer repetitions on a first pass, use caching for repeated identical calls, and set max_concurrency to stay under provider rate limits rather than having the sweep fail halfway.
Sourced from Docs by LangChain (MCP)
9When would you prefer a hybrid retriever with a reranker over plain dense retrieval?Very often asked
Dense retrieval handles paraphrase and semantic similarity well but can miss exact terms such as product codes, error strings or names. Keyword retrieval catches those but misses paraphrases. Hybrid combines both candidate sets, and a reranker reorders them by relevance to the question, often improving the top results. The trade-offs are extra latency, extra cost and more moving parts. So I don't assume; I include dense, keyword, hybrid, and each with and without a reranker in the grid, compare retrieval relevance and correctness, and look at latency in the traces. If the reranker's gain is within noise, the simpler pipeline wins.
Sourced from Docs by LangChain (MCP)
10How do you tell whether a score difference between two configurations is real?Very often asked
First, both experiments must use the same tagged dataset version and the same judge prompts and models; otherwise the comparison is invalid. Second, I use num_repetitions so each example runs several times and I can see the spread of scores per configuration. If the gap between variants is smaller than that spread, I treat it as noise. Third, I look per example in the comparison view: a real improvement tends to show up across many examples, while noise shows up as random flips. Finally, for close calls I send the two experiments to a pairwise annotation queue and let human preferences decide.
Sourced from Docs by LangChain (MCP)
11An answer scores high on correctness but low on groundedness. What does that mean and what do you do?Very often asked
It means the answer matches the reference but its claims are not supported by the retrieved documents, so the model probably answered from its own training knowledge or guessed. That is risky: on questions where the model's prior knowledge is wrong or outdated, the same behaviour produces confident hallucinations. I check retrieval relevance for that example. If the documents were irrelevant, retrieval is the problem. If relevant documents were retrieved but ignored, the generation prompt needs tightening, for example instructing the model to answer only from context. This is exactly why the promotion rule forbids regressing groundedness even when correctness improves.
Sourced from Docs by LangChain (MCP)
12How would you handle personal or sensitive data when turning production traces into dataset examples?Very often asked
Production traces can contain personal details, so they should not flow into a dataset automatically. I route candidates through an annotation queue where a reviewer checks each one, removes or redacts sensitive content, writes the reference answer, and only then exports it. I'd mask sensitive fields before or at tracing time where possible, restrict workspace access to people who need it, and keep datasets in the appropriate workspace. I also note that running an online evaluator upgrades a trace to extended retention, so I set filters and sampling rates deliberately rather than evaluating everything, and follow the organisation's retention and data-handling policy.
Sourced from Docs by LangChain (MCP)
13How do you design the retrieval relevance LLM-as-judge?Very often asked
It takes the question and the retrieved documents, not the answer, so it isolates the retriever. The prompt defines relevance clearly, for example whether the documents contain information needed to answer the question, and asks for a structured output: a score on a fixed scale or boolean, plus reasoning. Structured output keeps scores parseable and comparable across experiments; reasoning lets me audit disagreements. I start from a prebuilt template, then add few-shot examples taken from human-labelled cases. Before trusting it to pick a retriever, I compare its scores with human labels and fix the prompt where they disagree.
Sourced from Docs by LangChain (MCP)
14What do you put in experiment_prefix and metadata, and why does it matter?Often asked
The prefix gives each experiment a readable name, for example indicating the chunk size and retriever, so the comparison view is easy to scan. Metadata records the facts needed to reproduce and filter: chunk size, overlap, retriever type, reranker on or off, embedding model, generator model, prompt version, index identifier and dataset version tag. Without it, six months later nobody knows what an experiment actually tested. Metadata also lets me group and filter experiments when comparing a whole grid, and it is the evidence I cite when recording the promotion decision.
Sourced from Docs by LangChain (MCP)
15What happens if some target calls fail or time out during evaluate(), and how do you handle it?Often asked
Failed examples still appear in the experiment, but they have no valid output, so evaluators either error or score them poorly. If I just read the aggregate, a configuration with more timeouts could look worse for infrastructure reasons, or better if failures are silently skipped. So I check error counts per experiment first. Common causes are rate limits, which I address by lowering max_concurrency and adding retries with backoff in the target, and slow rerankers hitting timeouts. I rerun failed configurations cleanly rather than comparing partial experiments, and I treat a high failure rate as a real result about that configuration's reliability.
Sourced from Docs by LangChain (MCP)
16Why should the dataset version be tagged, and what do you do when the dataset needs to change?Often asked
Scores are only comparable on identical examples. If someone adds hard questions, the next experiment drops in score for reasons unrelated to the configuration. Tagging the benchmark version and pinning evaluate() to it keeps comparisons honest. Datasets should still evolve, since new failure modes appear in production. When I create a new version, I re-run the baseline and every candidate on it, and only compare experiments within that version. I never compare a variant on version two against a baseline on version one, which is exactly the mistake the workflow forbids.
Sourced from Docs by LangChain (MCP)
17How does chunk overlap affect RAG quality and cost?Often asked
Overlap repeats some text between adjacent chunks so that a fact split across a boundary still appears whole in at least one chunk. Too little overlap can break context and hurt retrieval relevance and correctness. Too much creates near-duplicate chunks, which grows the index, raises embedding cost, and can fill the top results with redundant passages, wasting context window and tokens. The right value interacts with chunk size and the retriever, so I test them as a grid rather than tuning one in isolation, and judge the outcome on the pinned dataset with repetitions.
Sourced from Docs by LangChain (MCP)
18How do you use the comparison view effectively?Often asked
I start with aggregates: correctness, groundedness and retrieval relevance per experiment, plus latency and token cost from traces. Then I filter to examples where configurations disagree, because averages hide trade-offs; one variant might win on long documents and lose on short factual queries. For each disagreement I open the trace to see the retrieved documents and the judge's reasoning, and check whether the judge itself made a mistake. That review often reveals dataset problems, like a wrong reference answer, which I fix in a new dataset version rather than silently.
Sourced from Docs by LangChain (MCP)
19How do you wire evaluation into pull requests?Often asked
I take key examples, especially past regressions and critical cases, and write them as pytest tests marked with @pytest.mark.langsmith. Running the test file logs a dataset and an experiment in LangSmith, so each pull request produces a traceable experiment I can compare against the main branch. I keep this suite small enough to run quickly and within budget, with thresholds on code checks and judge scores. The full configuration sweep stays a separate, scheduled or on-demand job. Vitest or Jest integrations do the same for TypeScript codebases.
Sourced from Docs by LangChain (MCP)
20What does online evaluation add after release, and how do you configure it?Often asked
Offline evaluation tells me how a configuration performs on a fixed dataset; online evaluation tells me how it behaves on real traffic, which drifts as users and documents change. I attach reference-free evaluators, such as groundedness and retrieval relevance, to production runs with filters and a sampling rate, since there is no reference answer in production. Dashboards and alerts flag drops, and automation rules can send low-scoring traces to an annotation queue, which then feeds new examples into the next dataset version. Sampling also controls cost and the extended retention that online evaluation triggers.
Sourced from Docs by LangChain (MCP)
21Which judge biases worry you in RAG evaluation, and how do you mitigate them?Often asked
Common ones are favouring longer or more confident answers, being lenient toward fluent text, and, in pairwise setups, preferring whichever answer appears first. A judge from the same model family as the generator may also be lenient to its own style. I mitigate by giving the judge narrow, explicit criteria per metric, requiring structured scores with reasoning, adding few-shot examples including tricky negatives, and checking agreement with human labels from annotation queues. For pairwise judgements I can swap answer order. If the judge disagrees with humans on a slice, I don't use it to decide on that slice.
Sourced from Docs by LangChain (MCP)
22Retrieval relevance is high but correctness is low. How do you debug it?Often asked
Good documents were retrieved, so the failure is probably downstream. I open the traces for failing examples. Possibilities: the relevant passage is ranked low and truncated from the context, the generation prompt is weak, the model misreads tables or numbers, or the reference answer itself is wrong. I also check whether the relevance judge is too lenient, marking documents relevant that only share keywords. I compare groundedness too: if the answer is grounded but incorrect, the documents may be outdated. Each cause has a different fix, which is why the metrics are kept separate.
Sourced from Docs by LangChain (MCP)
23How would you trace a RAG pipeline that is not built with LangChain?Often asked
LangSmith doesn't require LangChain. I set LANGSMITH_TRACING and decorate the pipeline functions, such as retrieve and generate, with @traceable so they appear as nested runs in one trace. For the model client I can use a wrapper like wrap_openai so token usage and latency are captured. If the service already uses OpenTelemetry, I can send spans to LangSmith's OTLP endpoint, and a collector can fan out the same telemetry to other backends. Either way, the target function for evaluate() produces traces that I can drill into from experiments.
Sourced from Docs by LangChain (MCP)
24Where do code-based evaluators fit alongside LLM-as-judge?Sometimes asked
Anything deterministic should be a code rule because it is cheap, fast and not noisy. Examples: the answer is non-empty, citations reference document IDs that were actually retrieved, output matches the required format, or the answer refuses when no documents are returned. Code rules can also check whether a known gold document appears in the retrieved set if the dataset stores it. LLM judges are reserved for semantic judgements like correctness against a reference or claim-level support. Combining both reduces cost and gives sanity checks on the judges.
Sourced from Docs by LangChain (MCP)
25How would you extend this workflow to a multi-turn conversational RAG assistant?Sometimes asked
Single-turn examples miss failures like losing context or misresolving follow-up references. In production, threads group multi-turn conversations, so I can review whole conversations and attach online evaluators at the thread level. Offline, dataset examples can carry conversation history in the inputs, with the reference answer for the final turn. Retrieval relevance must then judge documents against the rewritten, standalone question as well as the raw turn. For agentic variants where the assistant decides when to retrieve, agentevals trajectory evaluators can judge the sequence of steps.
Sourced from Docs by LangChain (MCP)
26What's the difference between reference-based and reference-free evaluators here, and why does it matter?Sometimes asked
Correctness is reference-based: it compares the answer with the dataset's reference answer, so it only works offline where references exist. Groundedness and retrieval relevance are reference-free: they use the question, answer and retrieved documents, so they can also run online on production traffic. This determines what I can monitor after release. It also means reference quality is critical for correctness; a wrong reference makes the judge penalise right answers, so references go through review and get fixed in new dataset versions.
Sourced from Docs by LangChain (MCP)
27What risks come with synthetic questions, and how do you control them?Sometimes asked
Synthetic questions generated from the corpus tend to copy its wording, so they are easier for keyword and dense retrievers than real user phrasing, which inflates retrieval scores. They can also be trivial, ambiguous, or have incorrect references. I review each before adding it, tag it with its source in metadata, and report scores by source so synthetic examples can't hide weakness on real questions. They are a supplement for coverage, not a replacement for curated and production-derived examples.
Sourced from Docs by LangChain (MCP)
28What would you record when you promote or reject a configuration?Sometimes asked
Enough for someone else to reproduce and audit it: the dataset version tag, baseline and candidate experiment names, the configuration metadata, aggregate scores with the spread across repetitions, latency and cost differences, the judge prompt versions and any human calibration results, pairwise annotation outcomes if used, and the reasoning, including why groundedness did not regress. I'd also note known risks and the online evaluators added to monitor it. That turns the choice from an opinion into evidence and makes rollback decisions faster.
Sourced from Docs by LangChain (MCP)
29How do prompts fit into versioning for this evaluation?Sometimes asked
Generation and judge prompts both affect scores, so they need versioning like code. I keep them in the Prompt Hub with commits and tags, and record the prompt commit or tag in experiment metadata. If a judge prompt changes, old and new experiments are no longer directly comparable, so I re-run the baseline with the new judge. Changing the generation prompt is its own variable; I avoid changing it at the same time as chunk size, otherwise I can't attribute the improvement.
Sourced from Docs by LangChain (MCP)
30How do you stop the evaluation sweep from becoming too expensive?Sometimes asked
I estimate calls first, then trim. Run a coarse grid with fewer repetitions to discard clearly weak configurations, then re-run the shortlist with full repetitions. Use code evaluators for deterministic checks so fewer judge calls are needed. Use caching where inputs are identical. Choose a judge model that is cost-effective but has been checked against human labels. Control max_concurrency to avoid rate-limit failures that force reruns. Online, use sampling rates instead of evaluating all traffic.
Sourced from Docs by LangChain (MCP)
31Why change only one dimension at a time, or use a full grid, rather than tweaking several things together?Sometimes asked
If I change chunk size, retriever and prompt in one go and scores improve, I can't tell which change helped or whether one change helped and another hurt. Crossing chunk size and overlap with retriever type in a defined grid, with everything else fixed, lets me attribute effects and spot interactions, such as a reranker helping only with small chunks. Metadata on each experiment makes the grid easy to slice. Unrelated changes like the generation prompt or model go into a separate comparison.
Sourced from Docs by LangChain (MCP)
32How would you calibrate the judges using the pairwise annotation queue?Sometimes asked
The pairwise queue contains exactly the two experiments, and reviewers choose which output is better per example, optionally with multiple reviewers per item and reservations so two people don't take the same item. I compare those human preferences with what the judge scores imply: if the judge prefers A where humans prefer B, I inspect the reasoning to find the misunderstanding. Disagreements become few-shot examples or clearer rubric wording in the judge prompt. After revising the judge, I re-check agreement before using it to decide again.
Sourced from Docs by LangChain (MCP)
33Would you ever skip the human review step?Rarely asked
Only when the decision is low risk and clear: the variant beats the baseline by well beyond the measured noise on the same dataset version, groundedness doesn't regress, and the judges have already been checked against human labels for this kind of data. Even then I spot-check disagreeing examples. For close results, surprising outcomes, new domains or a new judge prompt, human review is required. The workflow's rule is simple: no winner from noisy runs, mismatched dataset versions or an unchecked judge.
Sourced from Docs by LangChain (MCP)
Objective questions 33
1Why does the workflow build a separate index for each configuration?Asked almost every time
Why: Chunk size and overlap change the stored chunks, so a shared index would mix or retain stale chunks and invalidate the comparison.
Sourced from Docs by LangChain (MCP)
2Which evaluator compares the RAG answer against the dataset's reference answer?Asked almost every time
Why: Correctness is the reference-based evaluator; groundedness and retrieval relevance use the retrieved documents.
Sourced from Docs by LangChain (MCP)
3Which inputs does the retrieval relevance evaluator judge?Asked almost every time
Why: Retrieval relevance isolates the retriever by checking documents against the question.
Sourced from Docs by LangChain (MCP)
4Two experiments ran on different dataset versions and variant B scored higher. What is the correct conclusion?Asked almost every time
Why: Scores on different dataset versions are not comparable, so the workflow forbids declaring a winner from them.
Sourced from Docs by LangChain (MCP)
5Which inputs does a groundedness evaluator for a RAG answer fundamentally need?Very often asked
Why: Groundedness measures whether the answer's claims are supported by the retrieved context, so it needs the answer and the documents. It does not need a reference answer.
Sourced from Docs by LangChain (MCP)
6What should the @traceable target function return?Very often asked
Why: Groundedness and retrieval relevance evaluators need the documents as well as the answer.
Sourced from Docs by LangChain (MCP)
7Which evaluate() setting measures run-to-run noise?Very often asked
Why: num_repetitions runs each example several times so the variation can be measured.
Sourced from Docs by LangChain (MCP)
8Which evaluate() setting is used to manage provider rate limits?Very often asked
Why: max_concurrency controls how many calls run in parallel.
Sourced from Docs by LangChain (MCP)
9Under the promotion rule, a variant improves correctness but lowers groundedness. What happens?Very often asked
Why: A variant is only promoted if it improves the target signal without regressing groundedness.
Sourced from Docs by LangChain (MCP)
10What serves as the baseline experiment?Very often asked
Why: The workflow treats the current production configuration as the baseline.
Sourced from Docs by LangChain (MCP)
11What is the main reason to check an LLM-as-judge against human labels?Very often asked
Why: An unchecked judge may be biased or wrong, so winners can't be declared from it.
Sourced from Docs by LangChain (MCP)
12In LangSmith, how many experiments does a pairwise annotation queue compare?Often asked
Why: Pairwise annotation queues are built around exactly two experiments, and reviewers choose the better output for each example.
Sourced from Docs by LangChain (MCP)
13Why set repetitions when running evaluate() on RAG configurations?Often asked
Why: LLM outputs and LLM judges are non-deterministic. Repetitions show how much scores vary between runs, so small differences are not mistaken for real improvements.
Sourced from Docs by LangChain (MCP)
14Where should the experiment's chunk size and retriever type be recorded?Often asked
Why: The workflow passes metadata naming chunk size and retriever so experiments are reproducible and filterable.
Sourced from Docs by LangChain (MCP)
15Why should each judge return a structured score with reasoning?Often asked
Why: Structured output supports aggregation; reasoning helps explain disagreements.
Sourced from Docs by LangChain (MCP)
16How are production traces turned into reviewed dataset examples in this workflow?Often asked
Why: Single-run annotation queues let reviewers check traces and export them to datasets.
Sourced from Docs by LangChain (MCP)
17Which LangSmith feature turns test files into datasets and experiments for pull requests in Python?Often asked
Why: The pytest integration marks tests with @pytest.mark.langsmith so runs log as experiments.
Sourced from Docs by LangChain (MCP)
18What is a side effect of running an online evaluator on a trace?Often asked
Why: Per the docs, running an online evaluator upgrades a trace to extended retention.
Sourced from Docs by LangChain (MCP)
19Which evaluator cannot run on production traffic that lacks reference answers?Often asked
Why: Correctness compares with a reference answer, which production runs don't have.
Sourced from Docs by LangChain (MCP)
20An answer matches the reference but has claims unsupported by the retrieved documents. Which score is low?Often asked
Why: Groundedness checks that each claim is supported by the retrieved documents.
Sourced from Docs by LangChain (MCP)
21What does a reranker typically add to a retrieval pipeline?Often asked
Why: A reranker reorders retrieved candidates, improving ranking but adding latency and cost.
Sourced from Docs by LangChain (MCP)
22Where do you look to see latency and token cost per configuration?Often asked
Why: The workflow reviews latency and token cost from the traces in the comparison view.
Sourced from Docs by LangChain (MCP)
23Total judge calls in an offline sweep equal examples × configurations × repetitions × what?Sometimes asked
Why: Each target run is scored once by each LLM judge.
Sourced from Docs by LangChain (MCP)
24Which mechanism lets a non-LangChain service already emitting OpenTelemetry send traces to LangSmith?Sometimes asked
Why: LangSmith supports native OpenTelemetry via an OTLP endpoint, with fan-out through a collector.
Sourced from Docs by LangChain (MCP)
25What groups multi-turn conversations in LangSmith?Sometimes asked
Why: Threads group the runs of a multi-turn conversation.
Sourced from Docs by LangChain (MCP)
26Which tool provides trajectory evaluators for agentic RAG?Sometimes asked
Why: agentevals offers trajectory evaluators such as a trajectory LLM-as-judge.
Sourced from Docs by LangChain (MCP)
27Why are synthetic questions reviewed before entering the dataset?Sometimes asked
Why: Generated questions can copy corpus wording or contain incorrect references, distorting results.
Sourced from Docs by LangChain (MCP)
28How should online evaluators be applied to production runs to control cost?Sometimes asked
Why: Online evaluators support filters and sampling rates.
Sourced from Docs by LangChain (MCP)
29If a judge prompt is changed, what must happen before comparing with older experiments?Sometimes asked
Why: A different judge changes scoring, so comparisons require re-running under the same judge.
Sourced from Docs by LangChain (MCP)
30What is the main benefit of chunk overlap?Sometimes asked
Why: Overlap repeats boundary text so split context is preserved, at the cost of redundancy.
Sourced from Docs by LangChain (MCP)
31Which deterministic check is best implemented as a code evaluator rather than an LLM judge?Sometimes asked
Why: Exact ID matching is deterministic, cheap and not noisy.
Sourced from Docs by LangChain (MCP)
32Where are versioned generation and judge prompts managed with commits and tags?Rarely asked
Why: Prompts are versioned in the Prompt Hub and Playground with commits and tags.
Sourced from Docs by LangChain (MCP)
33Which feature in annotation queues prevents two reviewers working on the same item simultaneously?Rarely asked
Why: Annotation queues support reviewer counts and reservations.
Sourced from Docs by LangChain (MCP)
Track your progress through every blueprint in your dashboard.
Start learning