Skip to content
masterclassBuddyTech Buzz5 fresh blogs · 5:00 AM IST
Start learning

LangSmith blueprint · Tech & SaaS

Trace, evaluate, improve

LangSmith closes the loop: trace every run, turn real traffic into datasets and score each change.

Steps
7
Subjective questions
32
Objective questions
32

The problem and the outcome

Problem: LLM apps change behaviour silently whenever a prompt or model changes.

Outcome: Every change is measured against real data before it ships.

Architecture

LLM appany framework01Tracesruns · spans · tokens02Dashboards& alerts03Datasetexamples04EvaluatorsLLM-as-judge · code05Experimentcompare runs06Prompt / codefix07log runmonitorsave examplesscoresbest versionredeploy

How it works, step by step

  1. Turn on tracing (an environment variable or @traceable) and every run is logged.
  2. Each trace shows the full tree of model and tool calls, with inputs, outputs, latency and cost.
  3. Dashboards and alerts watch error rate, latency and user feedback in production.
  4. Save interesting traces as examples in a dataset.
  5. Evaluators score outputs: LLM-as-judge, exact match or your own code.
  6. Run an experiment for each change and compare the scores side by side.
  7. Ship the version that wins, and keep the loop running.

Tech stack

  • LangSmith SDK (Python / TypeScript)
  • LANGSMITH_TRACING and LANGSMITH_API_KEY
  • @traceable / traceable()
  • Datasets & experiments
  • Online evaluators, dashboards and alerts

Key design decisions

  • Trace everything from day one; the trace tree is how you debug an LLM app.
  • Build datasets from real production traces, not only invented examples.
  • Run an experiment for every prompt, model or code change and compare it with the baseline before shipping.

Common mistakes

  • Only checking a few examples by eye after each change.
  • Logging personal data in traces without masking it.
  • Evaluators that are never checked against human judgement.

What to measure

  • Evaluation scores per experiment
  • Error rate, latency and cost per run
  • User feedback score
  • Regressions caught before release

Most-asked interview questions

Subjective questions 32

1Why is observability harder for LLM apps than for normal services?Asked almost every time

Outputs are non-deterministic and free text, so 'did it work?' is a judgement, not a status code. A single request can involve many model and tool calls. You need the full trace of inputs, outputs, tokens, cost and latency per step, plus quality evaluation, not only uptime and error metrics.

2How would you set up evaluation for an LLM feature?Asked almost every time

Collect a dataset of representative inputs, with reference outputs where possible, from real traffic. Define evaluators: code checks for format, exact or semantic match against references, and LLM-as-judge with a clear rubric for qualities such as helpfulness. Run an experiment per change, compare against the baseline, check the judge against human labels, and keep adding failures to the dataset.

3Walk through the trace → evaluate → improve loop in LangSmith.Asked almost every time

Trace every run in development and production. Turn interesting runs (failures, low feedback, edge cases) into dataset examples. Run experiments on the dataset with evaluators for each prompt, model or code change. Compare experiments, ship the winner, and monitor production with online evaluators and dashboards, feeding new failures back into the dataset.

4How do you choose evaluators for an LLM feature?Asked almost every time

Use code evaluators for anything checkable (format, exact match, JSON validity, tool used), reference-based LLM judges for correctness against expected answers, reference-free judges for qualities like helpfulness or tone, and human review for subjective or high-stakes outputs.

5How do you detect regressions before release?Asked almost every time

Keep a versioned golden dataset, run it in CI for every change, compare scores against the production baseline experiment with thresholds per metric, and block releases when key metrics drop beyond noise.

6What is LLM-as-a-judge and what are its risks?Very often asked

Using a model to score another model's output against a rubric. It scales well, but judges can be biased toward longer or their own answers, inconsistent, or wrong on specialist topics. Mitigate with specific rubrics, reference answers, pairwise comparison, and regular calibration against human scores.

7How do you organise projects and environments?Very often asked

Use separate tracing projects per environment (dev, staging, prod) or tag runs with environment and version metadata, so dashboards and online evaluators run on the right traffic.

8How do you capture user feedback in production?Very often asked

Return the run ID to the client, record thumbs up or down and comments with create_feedback against that run, and filter traces by negative feedback to review and add to datasets.

9How do annotation queues fit into the loop?Very often asked

Send selected runs (low scores, sampled traffic, flagged feedback) to queues where reviewers label correctness and write corrected outputs, which become reference examples for datasets.

10How do you evaluate a multi-step agent in LangSmith?Very often asked

Score the final output, the trajectory (tool calls and order), and individual steps such as retrieval relevance, using datasets with expected outputs and trajectories and custom evaluators.

11How do you keep LLM-as-judge evaluators trustworthy?Very often asked

Give judges precise rubrics and examples, use a strong model at low temperature, check agreement with human labels on a sample, version judge prompts and re-validate when changing judge models.

12How do you manage prompts with LangSmith?Very often asked

Store prompts in the prompt hub with versions and tags (such as production), test new versions in the playground against datasets, and pull prompts by tag in code so updates do not need redeploys.

13How do you monitor cost and latency?Very often asked

LangSmith records tokens, cost estimates and latency per run and step; build dashboards per project and metadata (model, feature), and alert on spikes.

14How do offline and online evaluation differ in LangSmith?Very often asked

Offline evaluation runs your application on a dataset of examples, often with reference outputs, to create an experiment you can compare for benchmarking and regression testing. Online evaluation runs evaluators automatically on production runs or threads, which have no reference outputs, for real-time monitoring and alerting.

Sourced from Docs by LangChain (MCP)

15How do you keep the cost of online evaluators under control?Very often asked

Apply filters and a sampling rate so evaluators score only a fraction or a chosen subset of production runs. Note that when an online evaluator runs on any run in a trace, LangSmith upgrades that trace to extended data retention, which affects trace pricing.

Sourced from Docs by LangChain (MCP)

16What is the difference between a dataset example's inputs, outputs and metadata?Often asked

Inputs are what you send to the app, reference outputs are the expected results used by evaluators, and metadata tags examples (difficulty, source, category) for slicing results.

17How do you run an evaluation from code?Often asked

Call evaluate(target_function, data='dataset-name', evaluators=[...], experiment_prefix='v2') (Python) or the TypeScript equivalent; results appear as an experiment you can compare in the UI.

18How do you trace non-LangChain code?Often asked

Decorate functions with @traceable or wrap provider clients (such as wrap_openai), set the API key and tracing environment variables, and nested calls become a run tree.

19How do you protect sensitive data in traces?Often asked

Mask or hide inputs and outputs with configuration or custom anonymisers, avoid logging secrets, control project access, and set retention periods.

20How do you compare two prompts fairly?Often asked

Run both on the same dataset with the same evaluators and model settings, repeat runs to account for randomness, and look at per-example differences, not just averages.

21What are online evaluators?Often asked

Evaluators configured to run automatically on a sample of production traces, scoring things like relevance or toxicity without reference answers, feeding dashboards and alerts.

22How do automation rules help?Often asked

They act on traces matching filters, for example adding failed runs to a dataset, sending low-feedback runs to an annotation queue, or triggering online evaluators.

23How do you evaluate retrieval separately from generation?Often asked

Trace the retriever as its own step, create examples with the documents that should be retrieved, and score recall or relevance of retrieved chunks before judging the answer.

24What are the two kinds of annotation queue and when do you use each?Often asked

Single-run queues review one run at a time against rubric items, useful for triage and building datasets from production traces. Pairwise queues compare two runs side by side to judge which is better, designed for fast A/B comparison between two experiments.

Sourced from Docs by LangChain (MCP)

25How do you stop two reviewers judging the same item at once?Often asked

Enable reservations on the annotation queue, which hold an item for one reviewer for a set length of time. Queues also let you configure how many reviewers each run needs.

Sourced from Docs by LangChain (MCP)

26How do you turn production failures into regression tests?Often asked

Add failing production traces to a dataset, create targeted evaluators for the failure, validate a fix with an offline experiment on that dataset, then redeploy. This closes the loop from online monitoring back to offline evaluation.

Sourced from Docs by LangChain (MCP)

27How does a pairwise evaluation work on existing experiments?Often asked

Instead of scoring each output alone, you run evaluate() with two existing experiments and a pairwise evaluator that compares their outputs for the same example and scores which is better. Results are viewed side by side in the LangSmith UI.

Sourced from Docs by LangChain (MCP)

28How many examples should a dataset start with?Sometimes asked

Start with a few dozen well-chosen real examples covering main cases and known failures, then grow it continuously from production issues.

29What is a summary evaluator?Sometimes asked

An evaluator that computes one score over the whole experiment (such as precision, recall or F1 across all examples) instead of per example.

30How do you share evaluation results with stakeholders?Sometimes asked

Share experiment comparison links and dashboards, highlight example-level wins and losses, and summarise metric changes with their business meaning.

31Can LangSmith be used with OpenTelemetry?Sometimes asked

Yes, it accepts OpenTelemetry traces, so applications instrumented with OTel can send LLM spans to LangSmith alongside other tools.

32Which evaluator types does LangSmith support?Sometimes asked

Human review, code evaluators for rule-based checks, LLM-as-judge evaluators, and pairwise comparisons. Code evaluators can also be written in Python or JavaScript directly in LangSmith for online use, often to validate structure or statistical properties.

Sourced from Docs by LangChain (MCP)

Objective questions 32

1What is a trace in LangSmith?Asked almost every time
  1. AA billing invoice
  2. BA Git commit
  3. CA database backup
  4. DThe tree of every step (model and tool calls) in one runCorrect answer

Why: A trace records each nested run with inputs, outputs, timing and token usage.

2Where do new dataset examples best come from?Asked almost every time
  1. ARandom internet text
  2. BOnly synthetic data
  3. CMarketing copy
  4. DProduction failures and edge casesCorrect answer

Why: Real failures make the best tests.

3Which evaluator type should be used for checkable properties?Asked almost every time
  1. ANone
  2. BCode evaluatorsCorrect answer
  3. CLLM judges only
  4. DHuman review only

Why: Deterministic and cheap.

4What blocks a release in a regression gate?Asked almost every time
  1. AKey metrics dropping below the baselineCorrect answer
  2. BAny new example
  3. CA new prompt version
  4. DA long trace

Why: Compare to production.

5What does an experiment compare?Very often asked
  1. AServer CPU usage
  2. BGit branches
  3. CTwo databases
  4. DOutputs and scores of one app version over a dataset, against othersCorrect answer

Why: Experiments run a version over a dataset and score it, so versions can be compared side by side.

6How is user feedback linked to a trace?Very often asked
  1. ABy email
  2. BIt cannot be
  3. CRecord it against the run IDCorrect answer
  4. DBy timestamp only

Why: create_feedback with the run ID.

7Where do reviewers label and correct runs?Very often asked
  1. ADashboards only
  2. BAnnotation queuesCorrect answer
  3. CThe billing page
  4. DThe API key page

Why: Corrections become references.

8How should production code pick a prompt version?Very often asked
  1. ACopy text manually
  2. BUse the latest commit only
  3. CRandom version
  4. DPull by tag such as productionCorrect answer

Why: Updates without redeploys.

9What does an agent trajectory evaluation check?Very often asked
  1. ATool calls and their orderCorrect answer
  2. BUI layout
  3. CToken price
  4. DServer region

Why: Not just the final answer.

10What validates an LLM judge?Very often asked
  1. AAgreement with human labelsCorrect answer
  2. BIts speed
  3. CIts price
  4. DIts prompt length

Why: Calibrate on a sample.

11Which function runs an offline evaluation in Python?Very often asked
  1. Ajudge()
  2. Bscore_all()
  3. Cevaluate()Correct answer
  4. Dtrace()

Why: Results appear as an experiment.

12Online evaluators run on…Very often asked
  1. AProduction runs without reference outputsCorrect answer
  2. BOnly curated datasets
  3. CPrompt templates
  4. DBilling reports

Why: They monitor live traffic in real time.

Sourced from Docs by LangChain (MCP)

13What controls how many production runs an online evaluator scores?Very often asked
  1. AFilters and a sampling rateCorrect answer
  2. BThe model temperature
  3. CThe dataset size
  4. DThe number of reviewers

Why: Sampling keeps evaluation costs down.

Sourced from Docs by LangChain (MCP)

14How do you turn on tracing for a LangChain app?Often asked
  1. AInstall a browser extension
  2. BAdd a database trigger
  3. CRecompile LangChain
  4. DSet the LangSmith tracing and API key environment variablesCorrect answer

Why: Setting LANGSMITH_TRACING=true and an API key is enough for LangChain and LangGraph runs to be traced.

15What do online evaluators score?Often asked
  1. AOnly prompts
  2. BBilling
  3. CSampled production tracesCorrect answer
  4. DOnly datasets

Why: Usually without references.

16What can automation rules do?Often asked
  1. ATrain models
  2. BWrite code
  3. CDeploy apps
  4. DAdd matching runs to datasets or queuesCorrect answer

Why: Act on filtered traces.

17What is metadata on a dataset example used for?Often asked
  1. ARunning the model
  2. BEncrypting data
  3. CBilling
  4. DSlicing results by categoryCorrect answer

Why: Compare performance by segment.

18How do you trace a plain Python function?Often asked
  1. AWrap it in try/except
  2. BUse print
  3. CDecorate it with @traceableCorrect answer
  4. DRename it

Why: Nested calls become child runs.

19How should sensitive inputs be handled in traces?Often asked
  1. AMasked or hidden by configurationCorrect answer
  2. BLogged in full
  3. CEmailed
  4. DIgnored

Why: Masking protects user data.

20Comparing prompts fairly requires…Often asked
  1. ADifferent models
  2. BSame dataset, evaluators and settingsCorrect answer
  3. CDifferent datasets
  4. DOne run only

Why: Change only one thing at a time.

21How is retrieval evaluated separately?Often asked
  1. AOnly judge the answer
  2. BCount tokens
  3. CMeasure latency only
  4. DScore retrieved chunks against expected documentsCorrect answer

Why: Find whether retrieval failed.

22What do separate projects or tags per environment allow?Often asked
  1. AFaster models
  2. BFree usage
  3. CNo tracing
  4. DDashboards and evaluators on the right trafficCorrect answer

Why: Keep dev and prod apart.

23A pairwise annotation queue is created from…Often asked
  1. AExactly two experimentsCorrect answer
  2. BOne production project
  3. CA single prompt
  4. DAny number of datasets

Why: LangSmith pairs runs from the two experiments.

Sourced from Docs by LangChain (MCP)

24Reservations in an annotation queue…Often asked
  1. AHold an item for one reviewer for a timeCorrect answer
  2. BDelete reviewed items
  3. CReserve compute capacity
  4. DLock the dataset forever

Why: They prevent two people judging the same item at once.

Sourced from Docs by LangChain (MCP)

25What happens to a trace when an online evaluator runs on one of its runs?Often asked
  1. AIt is deleted
  2. BIt becomes a dataset
  3. CIt is anonymised
  4. DIt is upgraded to extended data retentionCorrect answer

Why: The upgrade affects trace pricing.

Sourced from Docs by LangChain (MCP)

26Running your app on a dataset to compare versions creates…Often asked
  1. AAn annotation queue
  2. BA prompt commit
  3. CAn experimentCorrect answer
  4. DA thread

Why: Experiments are compared for benchmarking and regressions.

Sourced from Docs by LangChain (MCP)

27A summary evaluator computes…Sometimes asked
  1. AOne score per token
  2. BA prompt
  3. CA dataset
  4. DOne score across the whole experimentCorrect answer

Why: For example F1 across all examples.

28A reasonable starting dataset size is…Sometimes asked
  1. AZero
  2. BA few dozen well-chosen examplesCorrect answer
  3. COne example
  4. DA million examples

Why: Grow it from production.

29Which standard can also send traces to LangSmith?Sometimes asked
  1. ASNMP
  2. BOpenTelemetryCorrect answer
  3. CSMTP
  4. DFTP

Why: OTel spans are accepted.

30Why repeat experiment runs?Sometimes asked
  1. ALLM outputs vary between runsCorrect answer
  2. BTo spend credits
  3. CIt is required
  4. DTo change data

Why: Averages are more reliable.

31What does LangSmith record per run for cost?Sometimes asked
  1. ANothing
  2. BTokens and cost estimatesCorrect answer
  3. COnly start time
  4. DOnly errors

Why: Build cost dashboards.

32Annotated runs from a queue can be…Sometimes asked
  1. ASent to customers
  2. BUsed as API keys
  3. CExported directly to datasetsCorrect answer
  4. DConverted into prompts automatically

Why: Reviewed runs become future evaluation examples.

Sourced from Docs by LangChain (MCP)

Track your progress through every blueprint in your dashboard.

Start learning