Customer support agent that resolves refunds

Put the policy in code: the refund tool checks eligibility, amount limits and order ownership itself, whatever the model asks for. Anything above a threshold or outside policy goes through a human-approval step. Log every refund with its trace and review samples.

Classify intent with a small model, look up the order and customer with read-only tools, check the refund policy with deterministic code (eligibility, window, amount), propose a resolution, route amounts above the auto-approve limit to a human via interrupt(), execute the refund with an idempotent tool, confirm to the customer and log the case.

Use the authenticated session's customer ID, never an order number alone typed in chat; for unauthenticated channels require verification (email OTP or order email match) before revealing details or acting.

Resolution rate without human help, correct refund decisions against audited samples, customer satisfaction, escalation rate, time to resolution, refund amount accuracy, cost per conversation, and complaints or reopened tickets.

Treat customer text as data. Tools enforce permissions and ownership server-side, the agent cannot reveal system prompts or other customers' data because its tools cannot fetch them, and suspicious requests are escalated. Test with adversarial examples in the evaluation set.

When confidence is low, the customer asks for a person, sentiment turns negative, the action needs approval, or the same issue has failed twice. The handoff should pass a summary and the trace, so the customer does not repeat themselves.

Encode eligibility (return window, item condition, category exceptions, amount caps) in code or a rules engine called as a tool; the model explains the result but cannot override it.

Make it idempotent with an order-specific key, enforce limits server-side, require approval context for large amounts, log every call with the conversation ID, and restrict it to the authenticated customer's orders.

Create a ticket with the conversation summary, verified customer and order details, what the agent tried and why it escalated, and tell the customer the expected response time.

Detect sentiment, respond calmly and briefly, prioritise escalation to a human for strong frustration, avoid arguing about policy, and apply abuse rules for threats.

Add focused read-only tools (track shipment, change address before dispatch, cancel unshipped order) behind the same authentication, with their own policy checks and approvals where needed.

Build a dataset of realistic conversations covering eligible, ineligible, edge and adversarial refund requests, run them against the graph with mocked commerce APIs, and score decisions, tool calls and tone, then pilot in shadow mode.

List allowed actions in the system prompt, ground promises in tool results (for example, only say a refund is issued after the tool confirms), and add an output check for unsupported commitments.

Keep structured fields separate from the message history: verified customer ID, order ID, order snapshot, detected intent, eligibility result, actions taken and their IDs, and escalation status. Tools read and write these fields, so decisions never depend on re-reading the chat.

Fetch the order lines, let the customer pick the item, check eligibility per line (return window, condition, final-sale flags), compute the amount including its share of discounts and tax, and call the refund tool with the line ID and an idempotency key.

Classification is easy and frequent, so a small model saves cost and latency; complex reasoning about the case uses a stronger model only when needed.

Detect the language, respond in it, keep policy logic language-independent in code, and test with native speakers for tone and accuracy.

Summarise older turns, keep order data in structured state rather than message text, and pass only relevant tool results.

Check customer refund history and velocity, block refunds for flagged accounts, require evidence (photos) for damage claims above limits, and escalate unusual patterns.

State clearly when the refund was issued and typical bank processing times by payment method, and send a confirmation email with the reference number.

Update the rules engine and its tests, update the policy summary used for explanations, rerun the evaluation dataset, and version the change with a release note.

Every decision with the policy rule applied, tool calls and results, human approvals with reviewer identity, and the final message to the customer.

Fetch past orders, refunds and tickets through tools, let rules consider them (for example loyal customers get goodwill credits), and avoid exposing other customers' data.

Explain the policy plainly and offer the closest allowed alternative, such as store credit or an exchange. If the case has unusual circumstances, hand over to a human with a summary instead of bending the rule in the bot.

Never answer from the model's memory: call inventory and shipping tools for live data, show timestamps or carrier status as returned, and say when information is unavailable rather than estimating.

Keep the agent logic channel-agnostic and add a speech-to-text and text-to-speech layer, shorten responses for speech, confirm critical details like order numbers by reading them back, and transfer to a person on the same call when needed.

Tag each conversation with intent, outcome and whether it was escalated, review a sample of failures weekly, cluster common reasons for escalation, and turn them into new tools, clearer policies or test cases.

Start with a small share of traffic or specific request types, monitor metrics and reviews closely, then expand once resolution quality matches human agents.

Tools return clear errors, the agent apologises and creates a ticket for follow-up rather than guessing, and alerts notify the team.

Ask a short satisfaction question at the end, link it to the trace, and review low scores to improve prompts, tools or policies.

Provide tone guidelines and examples in the system prompt, evaluate tone with an LLM judge on samples, and review a selection of conversations each week.

Trigger them from order events, not the chat agent, send through the notification system with the customer's preferred channel, and let a reply open a conversation with the agent preloaded with that order's context.

1. AIn the system prompt only
2. BIn the refund tool's codeCorrect answer
3. CIn the customer's message
4. DIn the vector store

Why: Code enforces the limit reliably; a prompt can be ignored or manipulated.

1. AOnly in the prompt
2. BBy the customer
3. CIn the CSS
4. DIn code or a rules engineCorrect answer

Why: The model cannot override code.

1. ANot at all
2. BFrom the authenticated session or verification stepCorrect answer
3. CBy trusting the typed order number
4. DBy asking the model

Why: Prevents acting on others' orders.

1. AAn idempotency key per orderCorrect answer
2. BA faster model
3. CLonger timeouts
4. DLogging only

Why: Retries must not double refund.

1. AIt is cheaper and faster for a simple classificationCorrect answer
2. BLarge models cannot classify
3. CIt avoids tracing
4. DIt is always more accurate

Why: Routing is an easy task, so a small model saves cost and latency on every message.

1. AAlways immediately
2. BNever
3. COnly after the tool confirms itCorrect answer
4. DWhen it plans to

Why: Ground promises in results.

1. AOnly the last message
2. BNothing
3. CThe full model prompt
4. DSummary, verified details and why it escalatedCorrect answer

Why: Humans should not re-ask.

1. AServer count
2. BResolution rate without human helpCorrect answer
3. CToken count
4. DPrompt length

Why: Pair with decision accuracy.

1. ANot at all
2. BRealistic and adversarial conversation datasetsCorrect answer
3. COnly by the developer chatting
4. DIn production only

Why: Then pilot it in shadow mode.

1. ARefund history and velocity checksCorrect answer
2. BFaster replies
3. CLonger conversations
4. DMore emojis

Why: Escalate unusual patterns.

1. APrioritised for human escalationCorrect answer
2. BArgued with
3. CIgnored
4. DBlocked

Why: Calm and brief responses.

1. AStructured state fields from toolsCorrect answer
2. BRe-reading the chat each time
3. CThe model's memory
4. DCustomer sentiment only

Why: Structured state is reliable and auditable.

1. AThe item's share of discounts and taxCorrect answer
2. BOnly the list price
3. CThe shipping carrier's name
4. DThe chat length

Why: Otherwise the customer is over- or under-refunded.

1. ASubgraphs
2. BReducers
3. CStreaming
4. Dinterrupt()Correct answer

Why: interrupt() saves the state and waits for a resume value from a person.

1. AIt is faster to type
2. BIt hides data from tools
3. CIt stays compact and reliable across turnsCorrect answer
4. DIt is required by HTTP

Why: Avoids bloated message history.

1. AThe policy rule applied and approvalsCorrect answer
2. BOnly the customer's name
3. CNothing
4. DThe model weights

Why: These records are needed for audits.

1. AExplain and create a follow-up ticketCorrect answer
2. BGuess the order status
3. CIssue refunds anyway
4. DEnd the chat silently

Why: The bot must never invent data.

1. ATell customers to wait
2. BRestart servers
3. CUpdate rules, tests and rerun evaluationsCorrect answer
4. DEdit the prompt only

Why: Version every policy change.

1. AOrder tracking
2. BAddress lookups
3. CGreeting
4. DDamage claims above limitsCorrect answer

Why: Reduces false claims.

1. ARandom dates
2. BOnly apologies
3. CWhen issued and typical bank timesCorrect answer
4. DNothing

Why: Set expectations clearly.

1. AIt is required
2. BClassification is easy and frequentCorrect answer
3. CSmall models are always better
4. DLarge models cannot classify

Why: Save cost and latency.

1. AHigher temperature
2. BShorter answers only
3. CNo system prompt
4. DGuidelines, examples and judge reviewsCorrect answer

Why: Review samples weekly.

1. AA refund anyway
2. BThe closest allowed alternative or a human handoverCorrect answer
3. CAn exception granted by the bot
4. DNo reply

Why: Bots must not bend policy on their own.

1. AThe product description
2. BLive shipping tool dataCorrect answer
3. CThe model's guess
4. DLast week's average

Why: Only live data is accurate.

1. ASpelled by the customer only once
2. BSent by email instead
3. CRead back for confirmationCorrect answer
4. DSkipped

Why: Speech recognition errors are common on numbers.

1. AChange the brand colours
2. BFind new tools, policies or test casesCorrect answer
3. CBlame agents
4. DDelete conversations

Why: Common failure reasons show what to build next.

1. AGradually by traffic share or request typeCorrect answer
2. BTo everyone at once
3. COnly at night
4. DNever

Why: Expand once quality matches.

1. ATo the refund tool
2. BTo the CDN
3. CLinked to the conversation's traceCorrect answer
4. DNowhere

Why: Review low scores to improve.

1. AIgnore policies
2. BApply goodwill rules for loyal customersCorrect answer
3. CShare data with others
4. DBlock everyone

Why: Within privacy limits.

1. AOnly the model's judgement
2. BA new app
3. CTheir own policy checks and authenticationCorrect answer
4. DNo checks

Why: They need the same safety bar.

1. ASummarisedCorrect answer
2. BDuplicated
3. CSent to the customer
4. DIgnored entirely

Why: Keeps context within limits.

1. AOrder events through the notification systemCorrect answer
2. BThe chat agent on a timer
3. CCustomers asking first
4. DManual emails only

Why: Events reach customers who never opened a chat.