Technology handbook
Start with individual concept cards in Agentic AI with LangChain and LangGraph, then explore Generative AI, RAG, Deep Agents, Guardrails, evaluation, gateways, design patterns, protocols, memory and production agents.
Memory is information an AI application retains for later use.
Example: A tutor remembers your preferred language while keeping today's exercise in conversation state.
Web search retrieves online information; deep search usually means a multi-step research process.
Example: Find a release announcement, read the migration guide, then compare both before recommending an upgrade.
A project groups related tasks, instructions and reference files into a workspace.
Example: Keep a product brief, terminology list and design rules together for a documentation project.
An artifact is a reusable output, such as code, a document or an interactive view, maintained separately from chat.
Example: Create a calculator interface, revise its validation and export the reviewed implementation.
HTML describes a web page's content and structure; CSS styles it and JavaScript adds behavior.
Example: Generate a FAQ page using headings and disclosure buttons, then test keyboard navigation.
AI document work includes extracting, summarizing, drafting and revising structured files.
Example: Turn meeting notes into an action list with an owner, due date and source note for each task.
AI-generated code is a proposed implementation that needs execution and review.
Example: Ask for a CSV parser, then verify quoted commas, empty rows and malformed input.
Claude Cowork is an agentic workspace for delegating tasks involving files and connected tools.
Example: Ask it to organize a permitted set of reports and prepare a summary; inspect the resulting files.
AI vocabulary connects model concepts, application architecture and operational controls.
Example: Describe an assistant as: a model reads context, requests tools, receives results and returns an evaluated answer.
A large language model learns patterns that let it assign probabilities to possible next tokens.
Example: After 'The capital of France is', the model may strongly favor 'Paris', but a plausible completion still needs verification when accuracy matters.
Prompting means supplying a model with a task, context, constraints and an expected output.
Example: Ask: 'Summarize this report in three bullets, preserve the figures and flag missing evidence.'
A system prompt supplies high-priority behavioral context in APIs that support that role.
Example: A support assistant is instructed to use supplied policy evidence and ask for clarification when an order is ambiguous.
A user prompt expresses the person's request and task-specific information.
Example: 'Compare these two proposals using cost, delivery time and maintenance effort.'
A developer prompt carries application-author instructions where the model API supports a developer role.
Example: An application requests a JSON object with a summary and evidence IDs for every user question.
Temperature controls sampling variability; the context window limits how much tokenized material a model can use in one request.
Example: For extraction, use a supported low-variability configuration and leave room for the response after adding the source text.
A hallucination is generated content that is false or unsupported despite sounding plausible.
Example: An assistant invents a package method; verify it against the installed package's documentation before using it.
Context engineering selects and organizes the information available to a model at each step.
Example: Before a refund decision, provide the relevant order, current policy and approval state instead of the entire customer database.
Tokens are the units into which a model's tokenizer encodes input and generated text.
Example: Measure a prompt with the target tokenizer instead of estimating cost from word count alone.
A vector is an ordered list of numbers; an embedding vector represents features of some input.
Example: A query about 'car repairs' can retrieve a passage about 'vehicle maintenance' without exact keyword overlap.
A context window is the supported token capacity available for a model request and its generation, subject to API limits.
Example: Keep the current task and recent tool results, summarize older discussion and reserve output capacity.
Parameters are learned numerical values used by a model to compute its outputs.
Example: Two models of similar size can perform differently because their training data, architecture and tuning differ.
Attention lets a model weight information from different token positions when forming representations.
Example: In 'The trophy would not fit because it was large', context helps connect 'it' with the relevant object.
A transformer is a neural-network architecture built around attention and feed-forward computation.
Example: A model uses the preceding paragraph to generate a coherent next sentence, one token at a time.
Training adjusts model parameters to reduce an objective measured over examples.
Example: A training run compares predicted tokens with target tokens and updates weights through optimization.
Fine-tuning continues training an existing model on selected examples for a narrower objective.
Example: Train on reviewed support-label examples, then measure classification quality on held-out tickets.
Reinforcement learning from human feedback uses human preferences to guide model behavior.
Example: Reviewers prefer helpful, supported answers over evasive or fabricated ones, supplying feedback for alignment training.
Retrieval-augmented generation supplies relevant external evidence to a model before it answers.
Example: Search the current employee handbook before answering a question about leave policy.
The Model Context Protocol standardizes how AI applications connect to external capabilities and context.
Example: An assistant uses an MCP server to discover and invoke an approved issue-tracking tool.
An AI agent uses a model to choose actions toward a goal within an application's control loop.
Example: An agent investigates a failed build, reads logs, proposes a patch and runs the permitted checks.
Multimodal AI works with more than one kind of information, such as text, images or audio.
Example: Ask a vision-capable model to describe a chart and compare its visible values with the accompanying text.
Reasoning models are designed or trained to spend additional computation on multi-step problems.
Example: Use a reasoning-capable model for a constraint puzzle, then independently verify the proposed solution.
Small language models use relatively fewer parameters than larger peers in a given comparison.
Example: Run a compact classifier locally to route messages, escalating difficult cases to a larger model.
AGI usually refers to broadly capable general intelligence; ASI refers to intelligence exceeding human capabilities across many domains.
Example: Assess a system's measured performance on tasks instead of inferring AGI from a convincing demonstration.
Chunking divides source material into smaller units for retrieval or processing.
Example: Split a manual by section and paragraph, then test whether real questions retrieve complete instructions.
An action is an operation that observes or changes an environment through an application capability.
Example: A calendar assistant distinguishes finding an available slot from actually creating the meeting.
Tools are functions an AI application exposes for specific operations beyond text generation.
Example: A weather tool accepts a location and returns measured forecast data for the assistant to summarize.
Structured outputs constrain a response to a defined shape, commonly a JSON schema.
Example: Extract an invoice into supplier, currency and total fields, then check that totals agree with line items.
An embedding model converts input into a numerical representation useful for matching or grouping.
Example: Embed support articles and a new question to find candidate passages for a RAG answer.
A vector database stores vectors and metadata and supports similarity-based retrieval.
Example: Retrieve the closest five permitted product-guide chunks for a customer's question.
An LLM API lets software submit inputs and receive model outputs over a defined interface.
Example: A backend receives a user's question, calls the model and streams the answer to the browser.
Zod defines TypeScript-friendly schemas that validate data at runtime.
Example: Validate that a tool's quantity is a positive integer before checking stock and creating an order.
Tool calling is the model requesting a named operation with structured arguments.
Example: The model requests lookupOrder with an ID; the server checks ownership and returns the matching order.
LangChain.js provides JavaScript and TypeScript components for model-powered applications and agents.
Example: Build a TypeScript assistant that can request a typed product lookup and explain the returned results.
LangGraph.js orchestrates stateful application steps using graph nodes and edges.
Example: Route a support ticket through classification, retrieval, draft review and a final response node.
Agent memory combines thread-scoped state with deliberately stored cross-session information.
Example: Resume a paused support conversation while retrieving only that user's saved preferences.
Human-in-the-loop design inserts human input or review into an automated process.
Example: Pause an uncertain invoice classification so an operator can select the correct department.
Agentic AI organizes model-driven decisions and actions into goal-directed applications.
Example: A researcher gathers evidence and a reviewer checks citations before a coordinator assembles the report.
Agents can automate parts of human work without becoming general replacements for people.
Example: Let an agent draft a hiring rubric while responsible staff review fairness and make employment decisions.
A workflow defines how a task progresses through steps, conditions and outputs.
Example: Receive a request, validate its fields, classify it with a model and route it using explicit rules.
Human approval authorizes a specific proposed operation before it is performed.
Example: Review the recipient and final email body before permitting the send operation.
An AI operator uses and supervises AI systems; an AI builder designs and implements those systems.
Example: An analyst runs a research assistant while an engineer maintains its retrieval and reporting tools.
An agent acts on a user's behalf when it executes an authorized operation to advance that user's goal.
Example: A travel assistant compares itineraries, then books only the itinerary authorized by the user.
Chatbots focus on conversational responses; agents can choose and execute steps toward a goal.
Example: A chatbot explains how to reset a setting; an authorized agent opens the relevant tool and changes it.
A core agent loop observes state, chooses a next step, acts and uses the result to decide what follows.
Example: Read a failing test, propose a fix, run the test again and stop when the acceptance criteria pass.
A prompt template combines stable instructions with named task-specific values.
Example: Fill a summarization template with a document and desired reading level while preserving a fixed output structure.
Debugging finds causes of failures; tracing records the sequence of model, tool and application steps.
Example: A trace reveals that retrieval returned an obsolete policy before the assistant generated the wrong answer.
A multimodal pipeline prepares text, images, audio or video for a model that supports those inputs.
Example: Summarize a lecture using its transcript and selected slides, citing the relevant timestamps.
An agent harness is the software environment around a model-driven execution loop.
Example: A coding harness supplies repository access, a test runner, a working directory and a record of completed changes.
You can build an agent application using an existing hosted or locally deployed model.
Example: Connect an available model to a read-only inventory tool and a review step to create a stock assistant.
Harness engineering improves the runtime capabilities and controls surrounding an agent.
Example: Add a bounded test runner and clear error results so a coding agent can repair a failure without unlimited retries.
Loop engineering is an informal term for designing the repeated decision-action-feedback cycle.
Example: Allow two repair attempts after a failed check, then return the diagnostic evidence for human review.
Graph engineering describes designing nodes, state and transitions for an application's control flow.
Example: Run independent research branches, merge their evidence and route disagreements to a review step.
Observability uses traces, metrics and logs to understand an application's behavior over time.
Example: Use Langfuse, LangSmith or Logfire instrumentation to identify a slow tool call within an agent run, with sensitive fields redacted.
Guardrails are layered checks and restrictions that reduce unwanted inputs, actions or outputs.
Example: Reject an unauthorized document lookup and scan the final answer for disallowed sensitive fields before returning it.
Evals measure application behavior against defined criteria; feedback records judgments or observed problems.
Example: Evaluate a support agent on answer correctness, citation support, prohibited tool use, latency and cost before releasing a prompt change.
Combine model and tool components with explicit state, routing and durable execution.
Choose fixed control flow when the steps are known and model-directed actions when necessary.
Validate a tool request, execute it under policy and return the result to the model.
Customize a standard agent using model, tool and context hooks.
Model shared state and define how updates merge across graph branches.
Persist progress and pause consequential actions for explicit review.
Delegate bounded work with evidence, ownership and a clear final synthesis contract.
Retain useful state while enforcing turn, time and spend limits.
Explain how a generative model produces outputs from learned patterns and supplied context.
Budget instructions, history, evidence and output within the model's supported context.
Specify a task, constraints and examples that make the desired response testable.
Turn model responses into typed data and reject invalid values before using them.
Combine text and supported media while respecting format and model capability limits.
Understand response variability and test repeated runs rather than trusting a single result.
Distinguish fluent generation from claims supported by verifiable evidence.
Compare prompting, retrieval and fine-tuning against a measurable task baseline.
Retrieve evidence before generation and separate retrieval quality from answer quality.
Preserve source IDs, versions and access permissions while extracting text.
Compare document-aware splits, chunk size and overlap using actual retrieval questions.
Index compatible vectors and keep the index synchronized with source changes.
Combine keyword and semantic recall, then rank a bounded evidence set.
Improve retrieval with focused subqueries without inventing assumptions about the user.
Tie each factual answer to accessible retrieved evidence and abstain when support is missing.
Measure retrieval recall and answer support while testing stale and deleted documents.
Retrieve evidence without a vector index using lexical search, structured data or document navigation.
Rank documents by lexical relevance and tune analyzers for the language and domain.
Answer from authorized database rows with validated queries and explicit result limits.
Use headings and document hierarchy to locate relevant sections before reading them.
Follow explicit entity relationships to collect evidence with bounded search depth.
Search filenames and content, then read selected passages under an access policy.
Pass a manageable authorized corpus to the model while tracking cost and omitted context.
Benchmark lexical, structural and vector retrieval on the same tasks instead of assuming a universal winner.
Use a fuller agent harness for context management, file access and delegated work.
Configure a Deep Agents application with an explicit model, tools and instructions.
Break complex goals into inspectable steps and revise the plan when evidence changes.
Distinguish virtual files from durable storage and choose a backend for the task lifetime.
Give each worker a focused task and return concise findings rather than all intermediate messages.
Move large results out of the active window while preserving retrievable references.
Separate model-generated code from privileged services and restrict network and filesystem access.
Test restart behavior and enforce resource limits across delegated work.
Combine validation, authorization, isolation and monitoring instead of relying on one filter.
Treat retrieved documents and tool output as untrusted data that cannot grant new permissions.
Check schemas, policy constraints and supported claims before progressing through the workflow.
Authorize the real caller and resource at the tool boundary regardless of the model's wording.
Prevent unnecessary personal data from entering model context and telemetry.
Show the exact proposed action and obtain a scoped decision before executing it.
Route uncertain cases through a defined policy without confusing moderation with factual verification.
Exercise malicious documents, privilege escalation attempts and data-exfiltration scenarios.
Define success on representative tasks before selecting a scoring mechanism.
Maintain reference examples with held-out cases and provenance for reproducible comparison.
Verify schemas, required facts and allowed actions with code where possible.
Use explicit rubrics and compare automated scores with human judgments.
Measure evidence selection separately from whether the final claims are supported.
Check both the actions taken and whether the requested result was actually achieved.
Compare prompt, model and graph revisions using the same dataset and evaluator definitions.
Turn production failures into test cases and prevent known regressions from shipping.
Place a controlled model-access layer between applications and provider endpoints.
Select backends through stable application-facing names with capability checks.
Recover from provider outages without duplicating work or creating retry storms.
Scope model access and spending independently for each application or tenant.
Control concurrency and token throughput before upstream quotas are exceeded.
Cache only suitable requests with keys that include user access and model configuration.
Track request outcomes and usage without collecting unnecessary sensitive content.
Check tools, streaming, structured output and errors before claiming provider interchangeability.
An agent alternates between reasoning about the next step, calling a tool, and reading the observation before deciding again.
Answering a question across live systems: A support agent asked "why was order 1042 delayed?" first looks up the order, sees it is waiting on a shipment, queries the carrier API with the tracking number, reads that the parcel is held at customs, and only then answers. Each lookup is chosen from what the previous one returned, which a fixed script could not anticipate.
Chain-of-thought reasons in one pass using only what the model already knows. ReAct interleaves reasoning with tool calls, so each step can use new external evidence, which reduces hallucinated facts on knowledge-heavy tasks.
Repeating the same call, drifting off the goal and running up cost. Contain them with a maximum step count, detection of repeated identical calls, a token or cost budget, and an explicit final-answer action the loop checks for.
A planner writes the whole multi-step plan first; an executor carries out each step, and the plan is revised only when a step's result requires it.
Researching and writing a market brief: Asked for a competitor brief, the planner lists: identify five competitors, collect each one's pricing page, extract plan tiers, compare features, draft the summary. Executors run each step; when one pricing page is behind a login, the planner swaps that step for the competitor's public docs instead of restarting the whole job.
When the task decomposes into mostly predictable steps and you want lower cost, a reviewable plan or parallel execution. ReAct suits exploratory tasks where the next step depends heavily on the last observation.
A failed step, a result that invalidates a later step's assumption, or new information that changes the goal. Replanning on every step throws away the cost advantage.
The agent reviews its own draft against explicit criteria, writes a critique, and revises — trading extra calls for higher-quality output.
Generating code that must pass tests: A coding agent writes a function, runs the unit tests, and feeds the failing assertions back as the critique. The revision targets the exact failures, and the loop ends when the tests pass or after three rounds, at which point the remaining failures are reported to a human.
A model often cannot spot its own factual errors without new evidence, so it may approve or even worsen a wrong draft. Grounding the critique in external checks — test results, retrieved sources, validators — is what makes reflection reliable.
Cap the rounds, only reflect on outputs that fail a cheap check, and use a smaller model for the critique where its accuracy is sufficient.
A task is split into a fixed sequence of model calls, each consuming the previous output, with programmatic checks between steps.
Producing a translated marketing email: Step one drafts the email from a product brief; a check confirms it mentions the required offer and stays under 150 words; step two translates the approved draft into Spanish; step three adapts tone for the regional audience. A failed check reruns only its step, not the whole chain.
It catches a bad intermediate result early, before later calls build on it, and gives a precise place to retry or fail — instead of discovering the problem only in the final output.
When the steps cannot be known in advance or vary per input; then routing or an agent loop fits better. Chaining also adds latency, so it is unnecessary when a single call already meets the quality bar.
A classifier step sends each input to the specialised prompt, tool set or model that handles that category best.
Triaging customer messages: Incoming messages are classified as billing, technical issue, refund request or general question. Billing goes to an agent with invoice tools, technical issues to one with log-search tools, refunds to a workflow that requires human approval, and general questions to a small, cheap model — so each path stays simple and well-tested.
Evaluate it as a classifier on a labelled set: accuracy and confusion between routes, plus the end-to-end quality of each route. A misroute often hurts more than a weak answer on the right route.
Most traffic is usually easy. Sending it to a smaller, cheaper model and reserving the expensive model for the hard category cuts average cost without lowering quality where it matters.
Independent subtasks run as simultaneous model calls (sectioning), or the same task runs several times and the results are combined (voting).
Reviewing a pull request: Three calls run at once: one checks security issues, one checks performance, one checks style against the team guide. Separately, the security check is run three times and a finding is reported only if at least two runs agree, which cuts false alarms without slowing the review.
It helps when individual runs are noisy and the cost of a wrong answer is high, such as safety or security checks. It wastes money when outputs are already stable or errors are systematic, since repeated runs then make the same mistake.
Sectioning is a fixed workflow: code decides the split and the merge. A multi-agent system lets models decide how to divide and coordinate work at run time.
A central model breaks a task into subtasks at run time, delegates each to a worker call, and synthesises their results.
Making a change across a codebase: Asked to rename a configuration option, the orchestrator searches for every file that uses it, then assigns one worker per affected module with the exact files and the new name. It collects their diffs, checks they are consistent, and runs the build once at the end.
Poor delegation: vague briefs make workers duplicate effort, miss parts or return incompatible results. Give each worker an objective, boundaries, the expected output format and the tools it may use.
Each worker sees only its subtask, and the orchestrator receives condensed results rather than every worker's full transcript, so no single context window has to hold the whole job.
One model generates a candidate while a second model evaluates it against clear criteria and returns feedback, repeating until the evaluation passes.
Translating legal text faithfully: A generator translates a contract clause; an evaluator compares it with the source and flags a dropped condition and a mistranslated term. The generator revises with that feedback, and the loop ends once the evaluator finds no meaning changes or after four rounds.
Reflection has the same agent critique itself; evaluator-optimizer gives judgement to a separate evaluator, often with its own model or rubric, which reduces the tendency to approve one's own work.
Explicit pass/fail criteria, a request for specific evidence for each issue, and a structured output (verdict plus feedback list) that the loop can act on programmatically.
An open protocol that lets AI applications connect to external tools, data and prompts through a standard client–server interface.
Giving an assistant access to company systems: A team publishes an MCP server for its ticketing system that exposes search_tickets and create_ticket tools plus ticket resources. The same server works in a desktop assistant, an IDE agent and an internal chatbot, and access is controlled in one place through the server's OAuth-based authorization.
Function calling defines how a model requests a tool inside one application. MCP standardises how tools and data are discovered and served across applications, so N apps and M integrations need N + M implementations instead of N × M.
A server can run code or reach data with the user's permissions, and tool descriptions or results can carry prompt injection. Use trusted servers, least-privilege credentials, user consent for sensitive actions and validation of tool output.
An open protocol for agents built by different vendors to discover each other, exchange messages and delegate tasks without sharing their internals.
Delegating across organisations: A travel company's planning agent finds a hotel partner's booking agent through its Agent Card, sends a task to hold a room for given dates, receives status updates while availability is checked, and gets a confirmation artifact back — without either side exposing its prompts, tools or models.
MCP standardises the agent-to-tool connection: a model calls tools and reads resources. A2A standardises agent-to-agent collaboration between independent, opaque agents that negotiate and complete tasks, potentially over long durations.
Partners can collaborate without revealing proprietary prompts, tools or data, and each agent stays free to change its implementation as long as it honours the advertised interface.
Agents use tools only as well as the tools are named, described and shaped, so tool interfaces deserve the same care as prompts.
Replacing a raw API with an agent-friendly tool: An agent struggled with three calendar endpoints (list users, list events, create event). Replacing them with one schedule_meeting tool that takes attendees and a time window, finds a free slot and returns a short confirmation — or a clear "no common slot this week" error — cut failed calls sharply.
Many overlapping low-level tools increase confusion over which to call, consume context with descriptions, and force the model to chain calls correctly. Task-shaped tools encode that chaining in code, where it is reliable.
A short, specific error saying what went wrong and what to try instead, such as an invalid date format with the expected format. A stack trace or a generic failure leaves the model guessing.
Agents that operate graphical interfaces by viewing screenshots and issuing mouse, keyboard or browser actions when no API is available.
Filing data into a legacy web portal: An insurer's claims system has no API. A browser agent in an isolated container logs in with a restricted account, fills the claim form from structured data, screenshots the confirmation, and pauses for human approval before the final submit button.
Prompt injection from on-screen content, unintended actions such as purchases or deletions, and credential exposure. Mitigate with sandboxed environments, allow-listed sites, minimal permissions and confirmation for irreversible steps.
They depend on visual layout, timing and pop-ups that change without warning, and each step is a perception task that can misread the screen; an API returns structured, stable data.
Agents that write and run code in a sandbox to compute, transform data or call many tools in one step, instead of issuing tool calls one at a time.
Analysing a large CSV export: Asked which region's revenue fell most last quarter, the agent writes Python to load a 200 MB export, group by region and quarter, and compute changes. Only the final ten-row table returns to the model, which explains the result — the raw data never enters the context window.
Filtering, joining and aggregating happen inside the sandbox, so only the result goes back to the model instead of every intermediate tool output passing through the context.
In an isolated container or VM with CPU, memory and time limits, a read-only or scratch filesystem, no production credentials and network egress restricted to approved hosts.
Packaged folders of instructions, scripts and resources that an agent loads on demand when a task needs that expertise.
Standardising report generation: A finance team packages its quarterly-report procedure as a skill: formatting rules, a chart script and a checklist. Any agent asked for a quarterly report loads that skill and follows the same procedure, while agents doing unrelated work never spend context on it.
A tool is an action the agent calls; a skill is knowledge about how to do a kind of task, which may include instructions, reference files and scripts the agent then runs with its tools.
Only skill names and descriptions sit in context until a skill is needed, then its full instructions load. It lets an agent have many capabilities without paying their context cost on every request.
Deciding exactly what enters the model's context at each step — instructions, tools, retrieved data, history — so it has what it needs and nothing that distracts it.
Keeping a long-running coding agent on track: After two hours, a coding agent's transcript holds thousands of lines of old test output. Instead of sending it all, each step supplies the task goal, the current plan, the files being edited, the latest test failures and a short summary of decisions so far — and the agent stays focused.
Prompt engineering focuses on wording instructions. Context engineering manages everything the model sees across a multi-step run — which documents, tool results, memories and history to include, compress or drop at each step.
Model accuracy degrades as the context grows long and cluttered, even within the window limit, because relevant details get lost among irrelevant ones. Curating context counters it.
Short-term memory is the working context of the current task; long-term memory persists selected facts across sessions in an external store.
A personal assistant that remembers preferences: During one chat a user mentions they are vegetarian and prefer morning meetings. The agent stores these as long-term memories tied to the user ID. Weeks later, booking a team lunch, it retrieves them and suggests a vegetarian-friendly place at 12:00 without being told again.
It grows without bound, costs tokens on every call, buries relevant facts and can exceed the context window. Extracting durable facts into long-term storage scales far better.
Per-user isolation, consent for what is stored, the ability to view and delete memories, retention limits and exclusion of sensitive data such as credentials.
A cognitive-science split for agent memory: episodes (what happened), facts (what is true) and procedures (how to do things).
An agent that improves at support tickets: A support agent stores successful resolutions as episodes and retrieves similar ones as few-shot examples (episodic), keeps product facts and customer plan details (semantic), and after repeated reviewer feedback updates its own escalation rules in its system instructions (procedural).
The agent rewrites part of its own instructions after feedback — for example adding "always confirm the account ID before issuing a refund" — so the behaviour persists in future sessions.
Embed past episodes and retrieve the most similar to the current task, then insert them as worked examples, filtered by user or tenant and recency.
Retrieval driven by an agent that decides whether, where and how often to search, and checks the results before answering.
Answering a multi-part policy question: Asked whether a contractor in Germany can expense a co-working space, the agent first searches the expense policy, finds that contractor rules live in a separate document, searches that, then queries the HR database for German-specific exceptions — and answers only once each part of the question has supporting evidence.
Standard RAG retrieves once with the user's query and answers. Agentic RAG puts retrieval in a loop the agent controls: choosing sources, rewriting queries, judging relevance and retrieving again until the evidence is sufficient.
Set a retrieval budget, require a relevance grade before continuing, and let the agent answer with stated uncertainty once the budget is spent.
Reusing the processed form of a repeated prompt prefix — system prompt, tool definitions, documents — so later calls cost less and respond faster.
Cutting the cost of a document-review agent: An agent reviews a 100-page contract over twenty tool-using turns. With the contract and tool definitions placed first and cached, each turn re-reads them at a fraction of the input price, and only the new turn's messages are processed in full — lowering both cost and latency for the whole run.
Put stable content first — tool definitions, system instructions, reference documents — and variable content such as the latest user message last, because only an identical prefix can be reused.
Anything that changes the prefix: a timestamp in the system prompt, reordered tools, or edits to earlier messages. Each change forces the prefix to be processed again at full price.
Recording every model call, tool call, input, output, latency and cost of an agent run as a trace, so failures can be found and fixed.
Debugging an agent that gives wrong refunds: Users report wrong refund amounts. The traces show the agent called the order tool correctly, but the tool returned prices in cents and the model read them as dollars. The team fixes the tool's output format and adds the failing trace to the regression dataset.
Agent failures arise from sequences of decisions. Flat logs lose the parent–child structure and the exact context the model saw at each step, which a trace preserves for replay and analysis.
Task success rate, steps and tool calls per task, latency, token cost per task, tool error rate, escalation or human-handoff rate, and user feedback.
Explicit limits on tokens, tool calls, time and spend per task so an agent's open-ended loop stays affordable and responsive.
Bounding a research agent: A research agent may use at most 15 searches, 200k tokens and 3 minutes per question. Query rewriting and summaries run on a small model; only the final synthesis uses the large one. When the search budget runs out, it answers from what it has and lists the open questions.
Cache stable prompt prefixes, route easy steps to smaller models, trim context to what each step needs, cap retries, and measure quality on an eval set after every change.
Stop cleanly, return the best partial result with a note on what is incomplete, and log the event, rather than failing silently or exceeding the limit.
Models that spend extra tokens reasoning before answering, with controls to set how much thinking a request may use.
Choosing effort per step: A data-pipeline agent uses low effort to rename columns and parse dates, but high effort when designing a schema migration that must preserve existing data — spending reasoning where mistakes are costly and saving it everywhere else.
For latency-sensitive, simple tasks such as classification, extraction or short replies, where extra thinking adds cost and delay without improving accuracy.
Give clear goals, constraints and success criteria rather than step-by-step instructions; the model plans the steps itself, and over-prescribing can reduce quality.
How much an agent may decide and do without a human — from suggesting actions, to acting with approval, to acting alone within limits.
Rolling out an IT-helpdesk agent: In month one the agent only drafts replies for staff. Once evaluations show 95% accuracy on password resets, it performs resets on its own, while account deletions and permission changes still need a technician's approval.
Weigh reversibility, blast radius and measured accuracy for that action: reversible, low-impact, well-evaluated actions can be autonomous; irreversible or high-impact ones keep a human approval step.
The agent recognises low confidence, missing permissions or out-of-policy requests, stops, and hands the case to a human with a summary of what it found and what it attempted.
Ask follow-up questions on any topic in your dashboard.
Start learning