LangSmith interview questions and answers

LangSmith is a platform for observing, evaluating and improving LLM applications. It records traces of every run (model calls, tools, retrieval, with inputs, outputs, tokens, cost and latency), lets you build datasets and run evaluations, compare experiments, review runs with people, and monitor production. It works with LangChain, LangGraph or any code through the SDK.

For LangChain and LangGraph, set LANGSMITH_TRACING=true and LANGSMITH_API_KEY (and optionally LANGSMITH_PROJECT), and runs are traced automatically. For other code, wrap functions with @traceable in Python or traceable() in TypeScript, or use wrappers for provider clients such as wrap_openai.

Create a dataset of examples (inputs and, ideally, reference outputs). Write a target function that runs your app on an input. Define evaluators that return scores. Call evaluate() (or client.evaluate) with the target, the dataset and the evaluators; the results appear as an experiment you can compare with others.

Start from real traffic: collect representative and difficult inputs from production traces, add known failures and edge cases, write reference outputs or criteria with domain experts, keep it versioned, and grow it whenever a bug is found. Fifty well-chosen examples beat thousands of random ones.

On each pull request, run evaluate() against a fixed dataset with deterministic and LLM-judge evaluators, compare scores with the main-branch baseline experiment, and fail the build if key metrics drop beyond a threshold. Post a link to the experiment comparison in the PR.

Offline evaluation runs your app on a curated dataset before release, giving comparable scores across versions. Online evaluation scores live production traces (sampled), catching real-world drift and issues your dataset missed, often without reference answers.

Code evaluators (exact match, regex, JSON validity, custom logic), LLM-as-judge evaluators with a rubric, pairwise evaluators that compare two experiments, and human review through annotation queues. Summary evaluators compute dataset-level metrics such as precision. Online evaluators run on production traces.

A queue of runs for people to review: they read the trace, give feedback scores or labels, and can correct the output and add it to a dataset. Automation rules can send runs to a queue, for example every run with negative user feedback.

Record the run ID when the app answers, then send feedback with client.create_feedback(run_id, key, score, comment) when the user clicks thumbs up or down. Feedback appears on the trace and can drive dashboards, filters and automation rules.

Online evaluators score a sample of live production traces automatically, for example for correctness or toxicity. Automation rules act on traces matching a filter: add them to a dataset, send them to an annotation queue, run an evaluator or trigger a webhook.

Run each as an experiment on the same dataset with the same evaluators, then use the comparison view to see score differences and the examples where the outputs differ. Pairwise evaluation helps for subjective qualities. Version the prompts in LangSmith so the winner can be promoted with a tag.

They label runs with information such as environment, app version, user tier, prompt version or experiment name, so you can filter traces, build dashboards per segment and compare versions in production.

Use the @traceable decorator (Python) or traceable wrapper (TypeScript) on functions, wrap provider clients such as wrap_openai, or send OpenTelemetry traces. Nested calls become child runs automatically.

The hierarchy of runs that makes up one trace: a root run for the request, with child runs for each chain, model call, retriever and tool. It shows inputs, outputs, timing, tokens and errors at every level.

A function that receives the run's inputs, outputs and reference outputs and returns a score (boolean or number) with an optional comment. It can use exact matching, regex, code execution or an LLM judge.

Filter traces by low feedback or failing online evaluators, group by tags such as version or model, open the run trees to find the failing step (often retrieval or a tool), add the cases to a dataset, fix and re-run the experiment to confirm.

Open a prompt or a traced model call, edit the prompt or model settings, and run it against examples or a whole dataset to compare outputs side by side, then save a new version to the prompt hub.

Collections of examples with inputs, optional reference outputs and metadata, used for evaluation and testing. Build them from real production traces, add edge cases and past failures, split them (for example by difficulty), and keep them versioned as they grow.

Save prompts in the prompt hub, where each change is a commit. Tag versions (such as prod or staging), test them in the Playground against datasets, and pull them in code by name and tag, so prompts can change without a redeploy.

Mask or hide inputs and outputs with SDK options or environment variables (for example hiding inputs/outputs entirely, or custom anonymisers), avoid logging secrets, restrict workspace access, and choose data retention settings or self-hosting where required.

Using a model to grade outputs against a rubric. Make it reliable with specific criteria, reference answers, asking for reasoning before the score, small score scales, few-shot examples of correct grading, and regular checks against human labels.

Sample traces with a rate setting, trace only certain environments, drop large inputs or outputs, and set data retention periods appropriately.

Use a clear rubric with examples, ask for reasoning before the score, use a strong model with temperature 0, keep the judge prompt versioned, and calibrate it against human labels on a sample.

Besides the final answer, check the sequence of tool calls: were the right tools used, in a sensible order, with valid arguments and no unnecessary steps. Compare with an expected trajectory or have a judge score it.

Set a number of repetitions so each example runs several times, then look at average scores and variance, since model outputs are not deterministic and one run can mislead.

Comparing two versions' outputs on the same inputs and asking which is better, by people or an LLM judge. It is useful when absolute scores are hard to define, such as for writing quality.

Request volume, error rate, latency (p50/p99), token usage and cost, feedback scores, online evaluator scores, and tool-call failures, filtered by model, prompt version or user segment.

No. The SDK traces any Python or TypeScript code and wraps popular provider clients, and OpenTelemetry can send traces too. LangChain and LangGraph just integrate automatically.

Share experiment comparison views, dashboards of production metrics and annotated example traces. Showing concrete good and bad examples beside the numbers builds trust.

Yes, there is a self-hosted option for enterprises that must keep trace data in their own infrastructure, in addition to the cloud service with regional data storage.

1. ALANGSMITH_TRACING=trueCorrect answer
2. BDEBUG=1
3. CTRACE_ALL=yes
4. DLANGCHAIN_VERBOSE=true

Why: With an API key set, LANGSMITH_TRACING=true traces LangChain and LangGraph runs automatically.

1. A@trace_me
2. B@traceableCorrect answer
3. C@observe_all
4. D@log

Why: @traceable records the function as a run in a trace.

1. AA new model
2. BResults of running an app version over a dataset with evaluatorsCorrect answer
3. CA billing plan
4. DA Git branch

Why: Experiments are what you compare across prompt or model versions.

1. ARandom internet text
2. BOnly synthetic data
3. CThe model's training data
4. DReal production tracesCorrect answer

Why: Real inputs reveal real failures.

1. AKey metrics drop below the baseline thresholdCorrect answer
2. BAny new trace
3. CA long prompt
4. DA new dataset

Why: Compare with the main-branch experiment.

1. Atrace_openai
2. Bopenai.trace()
3. C@openai
4. Dwrap_openaiCorrect answer

Why: Calls become child runs.

1. AAnnotation queuesCorrect answer
2. BThe billing page
3. CWebhooks
4. DThe CLI only

Why: Annotation queues collect runs for human review and labelling.

1. Aclient.create_feedback()Correct answer
2. Bclient.delete_run()
3. Cclient.new_project()
4. Dclient.pull_prompt()

Why: Feedback is attached to a run ID with a key and score.

1. AUnit tests
2. BOnline evaluatorsCorrect answer
3. COffline datasets only
4. DManual spreadsheets

Why: Online evaluators score sampled production traces.

1. ATwo datasets' sizes
2. BTwo outputs for the same inputCorrect answer
3. CTwo API keys
4. DTwo tags

Why: It judges which of two answers is better.

1. ABilling reports
2. BSampled live production tracesCorrect answer
3. COnly offline datasets
4. DPrompt templates

Why: It catches real-world drift.

1. AEncrypt data
2. BSpeed up models
3. CTrain models
4. DFilter and segment tracesCorrect answer

Why: For example by version or environment.

1. AA new prompt
2. BA dataset
3. CAn API key
4. DA score with optional commentCorrect answer

Why: Boolean or numeric.

1. AUI colours
2. BThe sequence of tool callsCorrect answer
3. COnly spelling
4. DToken prices

Why: Right tools, right order.

1. AIt is required
2. BTo change the dataset
3. COutputs vary between runsCorrect answer
4. DTo use more credits

Why: Averages are more reliable.

1. ADelete the workspace
2. BAdd them to a dataset or an annotation queueCorrect answer
3. CRetrain the model
4. DChange prices

Why: Rules route traces to datasets, queues, evaluators or webhooks.

1. AOnly the final answer
2. BA screenshot
3. CA single log line
4. DA tree of runs (spans) for each stepCorrect answer

Why: Nested runs show every model, tool and retriever call.

1. Atrace()
2. Bevaluate()Correct answer
3. Ctrain()
4. Ddeploy()

Why: evaluate() runs a target over a dataset with evaluators.

1. AFrom a screenshot
2. BFrom Git blame
3. CCopy-pasted text
4. DBy name and tag from LangSmith promptsCorrect answer

Why: Tags let you promote a tested version without a deploy.

1. AShorter prompts only
2. BHiding the reference
3. CA clear rubric and temperature 0Correct answer
4. DHigher temperature

Why: Also calibrate against humans.

1. AAnnotation queue
2. BBilling page
3. CProject settings
4. DThe Prompt PlaygroundCorrect answer

Why: Compare outputs side by side.

1. ASample tracesCorrect answer
2. BDisable all logs
3. CUse longer prompts
4. DAdd more tags

Why: Sampling keeps cost in check.

1. AOnly the final answer
2. BOnly errors
3. CBilling
4. DNested runs with inputs, outputs and timingCorrect answer

Why: Each step is a child run.

1. ALatency only
2. BUnit tests
3. CQuality that is hard to score absolutelyCorrect answer
4. DExact-match answers

Why: Ask which output is better.

1. AA queue
2. BA rubric
3. CA projectCorrect answer
4. DA dataset

Why: Traces are logged to a project, set by LANGSMITH_PROJECT.

1. AOnly Java apps
2. BOnly with LangFlow
3. CNo
4. DYes, through the SDK, client wrappers or OpenTelemetryCorrect answer

Why: It is framework-agnostic.

1. AA metric across the whole dataset, such as F1Correct answer
2. BToken prices
3. CThe model size
4. DA score per example only

Why: Summary evaluators see all results at once for dataset-level metrics.

1. ASOAP
2. BOpenTelemetryCorrect answer
3. CSMTP
4. DFTP

Why: OTel traces are supported.

1. AOnly on mobile
2. BYes, with self-hostingCorrect answer
3. CNo
4. DOnly in the EU

Why: A self-hosted option exists.

1. AIgnore it
2. BDelete the trace
3. CChange the model at once
4. DAdd it to the dataset as a test caseCorrect answer

Why: Datasets grow from failures.