Skip to content
masterclassBuddyTech Buzz5 fresh blogs · 5:00 AM IST
Start learning

LangGraph blueprint · Tech industry

On-call incident response agent

An investigator agent reads logs, deploys and runbooks, then proposes a fix for the engineer to approve.

Steps
7
Subjective questions
32
Objective questions
32

The problem and the outcome

Problem: At 3 a.m., the on-call engineer spends the first 30 minutes just gathering context.

Outcome: A cited root-cause hypothesis within minutes, and faster recovery.

Architecture

Pager alert5xx spike01Investigatoragent02Logs & metricsqueries03Recent deploys& diffsRunbooks &past incidents04Root-causehypothesis05Engineer approvesrollback06Postmortemdraft07resolved

How it works, step by step

  1. An alert fires: the error rate spikes on the checkout API.
  2. An investigator agent opens the incident and plans what to check.
  3. It queries logs and metrics and lists the deploys and config changes just before the spike.
  4. It retrieves the matching runbook and similar past incidents.
  5. It posts a root-cause hypothesis in the incident channel, with the evidence linked.
  6. The on-call engineer approves the suggested rollback; the agent never acts alone.
  7. After recovery it drafts the postmortem timeline for the team to edit.

Tech stack

  • LangGraph
  • Datadog / Grafana / Prometheus query tools
  • Deploy history (GitHub, Argo CD)
  • Runbook and postmortem retrieval
  • PagerDuty / Slack integration

Key design decisions

  • The agent investigates and proposes; an engineer approves any rollback or change.
  • Tools are read-only except one approved, reversible action (rollback) behind interrupt().
  • The hypothesis links the exact graphs, logs and diffs that support it.

Common mistakes

  • Agents that restart services on their own and make the incident worse.
  • Querying huge log volumes and blowing the context window.
  • Trusting a confident root cause without evidence.

What to measure

  • Mean time to acknowledge and to resolve (MTTA / MTTR)
  • Hypothesis accuracy confirmed in postmortems
  • Engineer time saved per incident

Most-asked interview questions

Subjective questions 32

1Design an on-call incident response agent.Asked almost every time

When an alert fires, the agent gathers context with read-only tools: recent deploys and config changes, error rates and latency by service, top log patterns, traces of failing requests and dependency health. It summarises likely causes with evidence links, suggests runbook steps, drafts status updates, and proposes mitigations such as rollback that require on-call approval.

2How do you keep the agent from flooding the model with telemetry?Asked almost every time

Query aggregated metrics and top-N log patterns rather than raw logs, compare to baselines, limit time windows, summarise each tool's output in code, and fetch details only when the agent asks for a specific service or error.

3Which actions can the agent take without approval?Asked almost every time

Read-only queries, posting summaries to the incident channel, creating the incident ticket and drafting status pages. Changes to production (rollback, scaling, feature flags, restarts) require an on-call engineer's approval.

4How would an incident agent gather context without flooding the model?Very often asked

Use targeted tools that return summaries: error counts by service and time window, top log messages, deploys in the last hour, the diff of the suspicious deploy. Filter and aggregate in the tools, not in the prompt, and fetch details only when the hypothesis needs them.

5What is MTTR and how can an agent reduce it?Very often asked

Mean time to recovery: the average time from an incident starting to service being restored. The agent shortens the investigation phase by collecting logs, metrics, recent changes and similar incidents in parallel within minutes, so the engineer starts from a hypothesis instead of a blank page.

6How do you correlate an incident with recent changes?Very often asked

Pull deploys, config changes and feature-flag flips from the last hours with timestamps and affected services, and compare their timing with when metrics degraded; most incidents follow a change.

7How do you use runbooks with the agent?Very often asked

Index runbooks with service and alert metadata, retrieve the matching one when an alert fires, let the agent follow diagnostic steps with tools, and suggest remediation steps for approval.

8How does the agent help with communication?Very often asked

It drafts status page updates and stakeholder messages in plain language with impact, scope and next update time, posts timelines to the incident channel and keeps notes for the postmortem.

9How do you evaluate an incident agent?Very often asked

Replay past incidents with recorded telemetry and check whether it identifies the root cause and the right runbook, measure time to useful hypothesis, and gather on-call engineers' ratings after real incidents.

10How do you protect against the agent making things worse?Very often asked

Read-only tools by default, approvals for changes, rate limits on actions, guardrails for blast radius (one service, canary first), and an easy way to turn the agent off.

11How does the agent support postmortems?Very often asked

It compiles a timeline from alerts, chat messages, deploys and actions, drafts the postmortem with impact, root cause and contributing factors, and suggests follow-up actions for the team to review.

12Which data sources does the agent need?Very often asked

Metrics (Prometheus, Datadog, CloudWatch), logs (Loki, Elasticsearch), traces (OpenTelemetry backends), deploy history (CI/CD), feature-flag systems, service catalogue with ownership and dependencies, and runbooks.

13How do you design the agent's investigation workflow?Very often asked

Start from the alert and affected service, gather recent deploys and config changes, query metrics and error rates for the service and its dependencies, pull sample error logs and traces, then rank hypotheses with evidence. Each step is a read-only tool call with a time budget.

14How do you decide when the agent should page a human?Very often asked

Page immediately for customer-facing SLO breaches, security signals and anything the agent cannot explain within a few minutes. Let the agent handle low-severity, known-pattern alerts with a summary, and always escalate if a remediation needs approval.

15Why keep a human approval step for rollbacks?Often asked

Rolling back can lose data migrations, conflict with other fixes or hide the real cause. The engineer has context the agent lacks, and accountability stays with a person. The agent prepares the exact command and its risks.

16What is MTTD vs. MTTR?Often asked

Mean time to detect is from failure start to alert; mean time to resolve (or recover) is from detection to service restored. Agents mainly reduce diagnosis time within MTTR.

17How do you avoid alert storms overwhelming the agent?Often asked

Group related alerts into one incident by service and time, deduplicate, and let the agent work on the incident rather than each alert.

18How do you handle secrets and access for the agent?Often asked

Give it a dedicated service identity with read-only scopes, short-lived credentials, audit logs of every query, and no access to customer data beyond what diagnostics require.

19How does the agent use the service dependency graph?Often asked

It checks upstream and downstream services for errors, identifies whether a dependency (database, third-party API) is the real source, and avoids chasing symptoms.

20Why keep a human approval for rollbacks?Often asked

Rollbacks can lose data, conflict with migrations or affect other teams; engineers weigh context the agent may lack and own the decision.

21How do you present hypotheses to on-call engineers?Often asked

List ranked hypotheses with evidence (graphs, log lines, deploy IDs), confidence, and the next check that would confirm or rule out each one.

22How do you keep runbooks useful over time?Often asked

Update them after each incident, link them to alerts, test them in game days, and let the agent report steps that were wrong or missing.

23What is an SLO burn rate alert?Often asked

An alert based on how fast the error budget is being consumed, which catches user-impacting problems early and reduces noisy threshold alerts.

24How does the agent help during a long, multi-team incident?Often asked

Maintain a live incident timeline from chat, alerts and actions, summarise the current state for people joining late, track owners and open tasks, and draft regular stakeholder updates for the incident commander to approve.

25How do you detect a bad deploy quickly?Often asked

Compare error rate, latency and saturation for the new version against the old during a canary, alert on significant differences, and let the agent link the anomaly to the specific deploy and suggest rollback for approval.

26How do you handle incidents caused by third-party providers?Often asked

Check provider status pages and your own synthetic checks against the provider, confirm whether errors come from their endpoints, switch to fallbacks or degraded modes where designed, and keep customers informed while waiting.

27How do you capture knowledge from incidents so the agent improves?Often asked

After each incident, store the confirmed root cause, signals that pointed to it and fixes that worked as structured data linked to the service. Update runbooks and add the incident as an evaluation case for the agent.

28How do you introduce the agent to an on-call team?Sometimes asked

Start in advisory mode in the incident channel, gather feedback, compare against manual diagnosis, and expand capabilities gradually as trust grows.

29What should be logged for agent actions?Sometimes asked

Every query, summary, suggestion, approval and executed action with who approved it and when, for postmortems and audits.

30What is a blast radius limit?Sometimes asked

A constraint so automated actions affect only a small scope at first (one region, one canary), limiting damage if the action is wrong.

31How do game days help?Sometimes asked

Practising simulated incidents tests runbooks, alerts and the agent's usefulness safely, revealing gaps before real outages.

32How do you prevent noisy, flapping alerts?Sometimes asked

Alert on symptoms users feel rather than every cause, use burn rates over sensible windows, add hysteresis so alerts do not fire and clear repeatedly, and review the noisiest alerts every month.

Objective questions 32

1What should the agent query instead of raw logs?Asked almost every time
  1. AEvery log line
  2. BRandom samples only
  3. CNothing
  4. DAggregated metrics and top log patternsCorrect answer

Why: Keeps context small and relevant.

2Which action needs on-call approval?Asked almost every time
  1. AReading metrics
  2. BPosting a summary
  3. CFetching runbooks
  4. DRolling back a deployCorrect answer

Why: Production changes need a human.

3Which change most often precedes incidents?Asked almost every time
  1. AA recent deploy or config changeCorrect answer
  2. BA new logo
  3. CA weekend
  4. DA blog post

Why: Correlate timing with changes.

4What does MTTR measure?Very often asked
  1. AMonthly traffic
  2. BAverage time to restore service after an incidentCorrect answer
  3. CModel token usage
  4. DDeploy frequency

Why: Mean time to recovery (or resolve) measures how quickly incidents are fixed.

5Mean time to resolve covers…Very often asked
  1. ATime from detection to service restoredCorrect answer
  2. BTime to write code
  3. CTime between deploys
  4. DServer uptime

Why: Agents shorten diagnosis.

6How should related alerts be handled?Very often asked
  1. AIgnored
  2. BDeleted
  3. CGrouped into one incidentCorrect answer
  4. DEach handled separately

Why: Grouping avoids alert storms.

7How is an incident agent evaluated?Very often asked
  1. AOnly in real outages
  2. BBy token count
  3. CBy reading its code
  4. DReplaying past incidents with recorded telemetryCorrect answer

Why: Did it find the root cause?

8What credentials should the agent use?Very often asked
  1. AAdmin keys
  2. BShared root accounts
  3. CA read-only service identity with short-lived tokensCorrect answer
  4. DA senior engineer's password

Why: Audit every query it makes.

9The dependency graph helps the agent…Very often asked
  1. AWrite tests
  2. BFind the real source instead of symptomsCorrect answer
  3. CDraw logos
  4. DDeploy faster

Why: Check upstream services.

10What does the agent draft for stakeholders?Very often asked
  1. AStatus updates with impact and next update timeCorrect answer
  2. BCode patches deployed automatically
  3. CMarketing emails
  4. DNothing

Why: Clear communication reduces confusion.

11Investigation tools used by the agent should be…Very often asked
  1. ARead-only with time budgetsCorrect answer
  2. BAble to delete data
  3. CUnlimited in runtime
  4. DShared admin accounts

Why: Investigation must not change production.

12The agent should page a human immediately for…Very often asked
  1. ADisk usage at 10%
  2. BSuccessful deploys
  3. CCustomer-facing SLO breachesCorrect answer
  4. DEvery low-severity alert

Why: Users are affected; people must be involved.

13Which change is most often linked to a sudden error spike?Often asked
  1. AA recent deploy or config changeCorrect answer
  2. BA new logo
  3. CA team meeting
  4. DDocumentation updates

Why: Most incidents follow a change, so recent deploys are checked first.

14Which action may the agent take without approval?Often asked
  1. ADeleting a database
  2. BScaling to zero
  3. CRolling back production
  4. DQuerying logs and metricsCorrect answer

Why: Read-only investigation is safe; changes need an engineer's approval.

15What is an SLO burn rate alert based on?Often asked
  1. ADisk space
  2. BNumber of deploys
  3. CHow fast the error budget is consumedCorrect answer
  4. DCPU only

Why: Catches user impact early.

16What does a blast radius limit do?Often asked
  1. ARestricts automated actions to a small scope firstCorrect answer
  2. BIncreases impact
  3. CDisables alerts
  4. DSpeeds up deploys

Why: Limits damage from mistakes.

17How should hypotheses be presented?Often asked
  1. AWithout evidence
  2. BAs raw logs
  3. CRanked with evidence and the next checkCorrect answer
  4. DAs one confident answer

Why: Engineers can verify quickly.

18Postmortem timelines come from…Often asked
  1. ACustomer emails only
  2. BAlerts, chat, deploys and actionsCorrect answer
  3. CMemory only
  4. DThe model's guess

Why: The agent compiles them.

19Which tracing standard is common for services?Often asked
  1. AFTP
  2. BMIDI
  3. COpenTelemetryCorrect answer
  4. DSMTP

Why: Vendor-neutral traces.

20MTTD measures…Often asked
  1. AAlert to fix
  2. BDeploy to release
  3. CCode review time
  4. DFailure start to alertCorrect answer

Why: It measures detection speed.

21How do runbooks stay useful?Often asked
  1. ADeleted
  2. BUpdated after incidents and tested in game daysCorrect answer
  3. CWritten once
  4. DKept secret

Why: Runbooks are living documents.

22How should the agent be introduced?Often asked
  1. AOnly after outages
  2. BAdvisory mode in the incident channel firstCorrect answer
  3. CFull automation immediately
  4. DHidden from engineers

Why: Trust grows gradually.

23During a long incident, the agent can…Often asked
  1. AClose the incident itself
  2. BSilence all alerts
  3. CMerge code fixes
  4. DKeep a live timeline and draft updatesCorrect answer

Why: It reduces coordination load on the commander.

24A bad deploy is detected fastest by…Often asked
  1. AReading commit messages
  2. BComparing canary and stable metricsCorrect answer
  3. CWaiting for customer emails
  4. DWeekly reports

Why: Side-by-side comparison shows regressions early.

25For a third-party outage you should first…Often asked
  1. AConfirm errors come from the provider's endpointsCorrect answer
  2. BRoll back your own code
  3. CRestart every server
  4. DDelete caches

Why: The fix depends on where the fault is.

26Resolved incidents should become…Often asked
  1. AEvaluation cases and runbook updatesCorrect answer
  2. BDeleted records
  3. CMarketing posts
  4. DNew alerts only

Why: Past incidents teach the agent and the team.

27What is a game day?Sometimes asked
  1. AA hackathon
  2. BA simulated incident practiceCorrect answer
  3. CA team party
  4. DA release day

Why: Tests runbooks safely.

28What must be logged for executed actions?Sometimes asked
  1. ANothing
  2. BOnly failures
  3. COnly the time
  4. DWho approved them and whenCorrect answer

Why: These records are needed for audits.

29Customer data access for the agent should be…Sometimes asked
  1. ALimited to what diagnostics requireCorrect answer
  2. BUnlimited
  3. CShared publicly
  4. DCopied to logs

Why: This is the least-privilege principle.

30How can engineers stop the agent quickly?Sometimes asked
  1. ADeleting the repository
  2. BCalling the vendor
  3. CAn easy kill switchCorrect answer
  4. DRebooting servers

Why: Safety comes before automation.

31Which metric shows the agent is useful?Sometimes asked
  1. ALines of logs read
  2. BModel size
  3. CPrompt length
  4. DTime to a useful hypothesisCorrect answer

Why: Faster diagnosis matters.

32Flapping alerts are reduced with…Sometimes asked
  1. APaging everyone
  2. BHysteresis and burn-rate windowsCorrect answer
  3. CMore alert rules
  4. DLower thresholds

Why: Alerts should not fire and clear repeatedly.

Track your progress through every blueprint in your dashboard.

Start learning