Food delivery: order to doorstep

The app sends the order with an idempotency key to the API gateway, which routes it to the order service. That service validates prices and opening hours, writes the order and an outbox event in one transaction, and starts a saga: authorise payment, then publish OrderPaid to Kafka. The restaurant service gets it and the restaurant accepts with a prep time; dispatch queries a Redis geo index for nearby riders, scores them and offers the order with a timeout. Riders' GPS updates the index, and a tracking service pushes live ETA over WebSockets while notifications fire on each status change. Failures run compensations: refund if the restaurant rejects, re-offer if a rider declines.

Send an idempotency key from the client on order creation and reuse it for the payment request, so retries return the same order and the same payment attempt. Store payment attempts with a unique constraint on that key, treat the gateway's signed webhook as the source of truth, and make the webhook handler idempotent because gateways retry. A daily reconciliation job compares the gateway's settlement report with your records and refunds any duplicates it finds.

Functional: browse restaurants and menus, build a cart, place and pay for an order, restaurant acceptance, rider assignment, live tracking, ratings, cancellations and refunds. Non-functional: high availability at meal-time peaks, order confirmation in about a second, no lost orders and no double charges, consistent order state across services, low-latency location updates, and graceful degradation when a partner (payment gateway, SMS provider) is slow.

User and auth, restaurant and menu catalogue, search, cart, order (the saga orchestrator), payment, restaurant-partner, dispatch, rider, tracking, notification, pricing and promotions, and ratings. They change at different speeds, scale differently (search and tracking are read- or write-heavy at peak) and are owned by different teams, so splitting them lets each scale and deploy on its own.

An order touches payment, the restaurant and dispatch, which are separate services and often external partners that cannot join a two-phase commit. Holding locks across them for seconds would also kill throughput. A saga runs each step as a local transaction and defines a compensating action for each (refund, cancel, release rider), so the system ends consistent without global locks.

Orchestration: the order service holds the state machine (PENDING → PAID → ACCEPTED → ASSIGNED → PICKED_UP → DELIVERED) and tells each service what to do next. The flow is easy to read, change, monitor and time out in one place. Choreography, where services react to each other's events, suits simple flows but becomes hard to follow as steps and compensations grow.

Writing to the database and then publishing to Kafka are two separate operations; a crash between them loses the event, or a publish before a rollback announces an order that never existed. With an outbox, the order row and an event row are written in one local transaction, and a relay (polling or change-data-capture such as Debezium) publishes outbox rows to Kafka. Consumers must still be idempotent, because the relay can publish an event more than once.

Verify the signature with the gateway's secret, look up the payment by its order or payment ID, and process it idempotently: store the event ID with a unique constraint so a retried webhook is a no-op. Respond 200 quickly and do the real work asynchronously. Never trust the client's redirect as proof of payment, and if a webhook is late, query the gateway's status API before cancelling.

Riders send GPS every few seconds to a location service that writes the latest point to a geo index: Redis GEOADD keyed by city, or riders bucketed into H3 or geohash cells. Entries carry a short TTL or a last-seen time, so offline riders drop out. Dispatch queries by radius around the restaurant with GEOSEARCH. A sample of pings goes to a stream for analytics and ETA models, never to the primary orders database.

Shard dispatch by city or zone, because a match never crosses zones. Within a zone, collect new orders over a short window (a few seconds) and solve the assignment as a batch, scoring rider–order pairs by pickup ETA, prep time, load and fairness, which beats greedy one-by-one matching at peak. Keep a lock per order and per rider so one offer is outstanding at a time, and re-run matching for declined or timed-out offers.

Keep live rider positions in a geo index (Redis GEOADD with short TTLs, or H3 cells). On OrderPaid, query riders within a radius of the restaurant, filter by status and vehicle, then score by pickup ETA, the restaurant's remaining prep time, current load and fairness. Offer to the top rider with a 20–30 second timeout and a lock on the order, re-offering to the next on decline. At peak, batch orders from the same restaurant and assign in short windows to optimise globally rather than greedily.

The app retries with the same idempotency key, so the server returns the existing order and payment attempt instead of creating new ones. The order stays PENDING until the gateway's webhook or a status poll says success or failure. If nothing arrives within the timeout, a background job queries the gateway; a captured payment moves the order on, and a failed or abandoned one cancels it.

ETA = time until food is ready (from the restaurant's prep time and history) + rider travel to the restaurant + travel to the customer, using a routing engine and live traffic, corrected by a model trained on past deliveries. Recalculate on every status change and on fresh GPS, smooth the number so it does not jump around, and show a range rather than a single minute when confidence is low.

The app opens a WebSocket (or Server-Sent Events) connection to a tracking gateway once the order is assigned. The gateway subscribes to that order's location updates (Redis pub/sub or a Kafka topic keyed by order) and pushes position and ETA every few seconds. Connections are spread across gateway nodes behind a load balancer, and the app falls back to polling if the socket drops.

The restaurant service starts a timer when it delivers the order to the partner app, with reminders such as a ring or a call from an automated system. If there is no acceptance within the limit, the saga cancels the order and refunds automatically, and the restaurant may be marked temporarily unavailable so new customers do not order from it.

Make assignment a compare-and-set on the order: set the rider only if the order is still unassigned (a conditional update or a Redis lock with an expiry), and offer to one rider at a time with a timeout. A rider's accept call must also pass that check, so a late acceptance after a re-offer fails cleanly and the rider is told the order is gone.

Menus change rarely but are read constantly, so serve them from a cache and a CDN-backed API, with the catalogue service as the source of truth. Restaurants publish changes that invalidate or update cached menus through events. Availability (out of stock, closed) is a small, fast-changing overlay checked again at checkout, so a stale cache can never sell an item that is gone.

Auto-scale the stateless services ahead of known peaks, keep the hot reads (menus, search) on caches and CDNs, and queue the work that can wait (emails, analytics). Kafka absorbs bursts between services. Shed load gracefully: rate-limit per user, switch off expensive features such as personalised recommendations, and widen the dispatch batching window instead of failing orders.

A relational database such as PostgreSQL for orders and payments, because they need transactions and constraints (unique idempotency keys, valid state transitions). Partition or shard by order ID or by city as volume grows, keep recent orders hot and archive old ones, and serve history and analytics from read replicas or a warehouse rather than the primary.

One owner: the order service holds the state machine and is the only writer of the order's status. Other services emit events (PaymentCaptured, OrderAccepted, RiderAssigned) and the order service applies valid transitions only, rejecting impossible ones such as DELIVERED before PICKED_UP. Events carry the order ID and a version, so out-of-order or duplicate events are detected and ignored.

Cancellation is a saga step whose effect depends on the state: before acceptance, a full refund; after the restaurant starts cooking, a partial or no refund, per policy; after pickup, usually not allowed. The order service records the cancellation, runs the matching compensations (refund, notify restaurant, release the rider) and each step is idempotent, so a retried cancel does nothing twice.

Index restaurants and dishes in a search engine (Elasticsearch or OpenSearch) with geo fields, cuisine, rating, price and availability, and filter by delivery radius first. Rank by relevance, distance, ETA, rating and business rules. Keep the index updated from catalogue events, cache popular queries per area, and fall back to a simpler list if search is degraded.

Business signals first: orders per minute against the forecast, payment success rate, restaurant acceptance time, time to rider assignment and on-time delivery. Then system signals: p99 latency per service, Kafka consumer lag per topic, WebSocket connection counts, error rates and dead-letter queue growth. Alert on the business signals, because a drop in orders per minute catches problems that no single service metric shows.

Run a reconciliation job that matches the gateway's settlement and refund reports with your payment and order records by ID and amount. Flag mismatches: captured but no order, order paid twice, refund not settled. Fix them automatically where it is safe (refund a duplicate) and send the rest to an operations queue. This catches webhook losses and bugs that idempotency alone cannot.

Kafka delivers at least once, so each consumer records the event IDs it has processed (or uses a natural key with a unique constraint) and skips duplicates. Side effects are idempotent: 'set status to ACCEPTED' rather than 'toggle', 'refund for payment X' with a unique refund key. Poison messages go to a dead-letter topic after a few retries so they never block the partition.

Load-test with replayed and synthetic traffic at several times the expected peak, including payment-gateway mocks with realistic latency and failures. Run chaos experiments: kill a dispatch node, delay webhooks, add Kafka lag, and check that sagas compensate and nothing is lost. Practise the runbooks, pre-scale, and freeze risky deploys for the event.

Status changes publish events; a notification service consumes them, picks the channel (push first, SMS for critical steps or when push fails), applies user preferences and templates, and sends asynchronously through providers. Failures are retried with backoff and dead-lettered. Nothing in the order path waits for a notification to be delivered.

A pricing service computes the fee from distance, basket value, demand and rider supply in the zone, and promotions. The quote is calculated at checkout, signed or stored with an expiry, and honoured when the order is placed, so the customer pays what they saw. Surge rises gradually with the demand-to-supply ratio and is capped, with the rules logged for audit.

The rider's last position stays visible with a 'last updated' time, and the ETA widens. Dispatch watches for a missed heartbeat; after a threshold, operations are alerted and the customer is told. If the rider reconnects, the app syncs queued status updates (with timestamps) idempotently. If not, support can reassign or contact the rider.

When two orders come from the same restaurant or lie along one route, dispatch can assign them to one rider if both promised ETAs still hold. The matching score includes the detour for each customer, a cap of two or three orders, and food-quality limits on extra minutes. Each order keeps its own tracking and status even though the rider is shared.

Eventual consistency between services in exchange for availability and independent scaling, so the UI must show in-between states. Batch dispatch gives better matches but adds seconds of delay. Caching menus is fast but needs an availability check at checkout. Sagas avoid global locks but need careful compensations and idempotency everywhere. Good answers name the trade-off and why it is acceptable here.

1. AA cron job that scans all orders
2. BTransactional outboxCorrect answer
3. CTwo-phase commit across Kafka and the database
4. DPublishing before the database write

Why: The event is written to an outbox table in the same transaction as the order, and a relay publishes it, so the two can never disagree.

1. AAsk the customer to call support
2. BRun the compensating action: refund the payment and cancel the orderCorrect answer
3. CRoll back a distributed transaction
4. DLeave the payment and retry the restaurant forever

Why: Sagas undo completed steps with compensating actions; here that means an automatic refund.

1. ASo a retried request returns the same order instead of creating a duplicateCorrect answer
2. BTo encrypt the order
3. CTo choose the nearest rider
4. DTo speed up the database

Why: Networks fail after the server has acted; the key lets the server recognise the retry and return the first result.

1. AThe customer's app
2. BThe order service's state machineCorrect answer
3. CThe payment gateway
4. DKafka itself

Why: Orchestration puts the flow in one service, which reacts to each result and issues the next command or compensation.

1. AWritten in Java
2. BIdempotentCorrect answer
3. CSingle-threaded
4. DStateless and without storage

Why: Duplicates are expected with at-least-once delivery, so processing the same event twice must have no extra effect.

1. ALPUSH
2. BGEOSEARCHCorrect answer
3. CZRANGE
4. DHGETALL

Why: GEOSEARCH (the successor to GEORADIUS) queries members of a geo set by radius or box around a point.

1. AIgnore it
2. BRefund the paymentCorrect answer
3. CDelete the payment record
4. DCharge again

Why: Compensation undoes the business effect of a completed step; for a capture that is a refund, recorded with its own key.

1. AIt removes the need for consumers
2. BIt encrypts each order
3. CAll events for one order stay in sequence in one partitionCorrect answer
4. DIt makes messages smaller

Why: Kafka guarantees order only within a partition, so keying by order ID keeps each order's events in order.

1. AThe gateway's signed webhook or its status APICorrect answer
2. BThe browser redirect back to the app
3. CThe customer's screenshot
4. DThe cart total

Why: Redirects can be faked or lost; only a verified webhook or a server-side status check confirms the payment.

1. AThe search engine's menu index
2. BAn in-memory geo index such as Redis with short TTLsCorrect answer
3. CThe primary orders table
4. DA CDN

Why: Locations change every few seconds and only the latest matters, which suits an in-memory geo index.

1. ASending the offer twice
2. BLetting the faster rider win without checks
3. CCaching the order
4. DA conditional update or lock that assigns the order only if it is still unassignedCorrect answer

Why: A compare-and-set makes the second acceptance fail cleanly instead of double-booking.

1. AAssigns riders
2. BPublishes event rows from the outbox table to the message brokerCorrect answer
3. CDeletes old orders
4. DCharges the customer

Why: It moves committed events to Kafka, by polling or change-data-capture, decoupling the write from the publish.

1. AIt is always faster per order
2. BIt needs no geo index
3. CIt avoids ETAs
4. DIt considers many orders and riders together, giving better overall assignmentsCorrect answer

Why: Greedy assigns the nearest rider now and can leave later orders badly served; a short batch optimises the whole zone.

1. ACharge a fee to the customer
2. BCancel the order and refund automaticallyCorrect answer
3. CWait forever
4. DAssign a rider anyway

Why: A timeout is a failure of that step, so its compensations run and the customer is refunded.

1. AThe CDN is down
2. BDispatch cannot process events as fast as they arriveCorrect answer
3. CKafka is losing messages
4. DOrders are being refunded

Why: Lag is the gap between the latest offset and the consumer's position; growth means consumers need to scale or are stuck.

1. AThe order service that owns the state machineCorrect answer
2. BEvery service that touches the order
3. CThe customer's app
4. DThe notification service

Why: A single owner applies only valid transitions, so services cannot leave the order in contradictory states.

1. ASlow menus
2. BRider fraud only
3. CSearch ranking errors
4. DLost webhooks and mismatches between the gateway's records and yoursCorrect answer

Why: Comparing settlement reports with your records finds payments that never reached your system or were recorded wrongly.

1. AReject the order
2. BRecognise the event ID and do nothing the second timeCorrect answer
3. CCapture the payment again
4. DRefund it

Why: Gateways retry webhooks, so storing the event ID with a unique constraint makes the duplicate a no-op.

1. AWebSockets (or Server-Sent Events)Correct answer
2. BSMTP
3. CFTP
4. DBatch CSV upload

Why: A persistent connection lets the server push updates every few seconds without the app polling.

1. AAn in-memory cache only
2. BA search engine
3. CObject storage
4. DA relational database with transactions and unique constraintsCorrect answer

Why: Orders need atomic writes, valid states and unique idempotency keys, which relational databases enforce.

1. AStore them only on the phone
2. BNever update them
3. CCache them, invalidate on change, and re-check availability at checkoutCorrect answer
4. DRead them from the primary database on every view

Why: Caching serves the reads cheaply; the checkout check protects against selling an item that just ran out.

1. ANumber of menu items
2. BCPU of the search cluster
3. CApp store rating
4. DTime from payment to rider assignmentCorrect answer

Why: Slow assignment is the first sign that riders are short or matching is overloaded.

1. AIt should be deleted silently
2. BTo the customer
3. CA dead-letter topic for inspection and replayCorrect answer
4. DBack to the front of the same partition forever

Why: A dead-letter topic stops one bad message from blocking the partition while keeping it for later.

1. AIt hides data from riders
2. BIt removes the need for maps
3. CIt makes ETAs exact
4. DA rider is never matched to an order in another zone, so zones scale independentlyCorrect answer

Why: Matching is local, so splitting by zone divides the load without cross-zone coordination.

1. AAdds a member with longitude and latitude to a geo setCorrect answer
2. BDeletes a key after a time
3. CPublishes a message
4. DIncrements a counter

Why: GEOADD stores positions in a sorted set encoded for geo queries such as GEOSEARCH.

1. ATo avoid templates
2. BSo slow providers never delay the order flow, and failures can be retriedCorrect answer
3. CBecause push is illegal otherwise
4. DTo make them arrive in random order

Why: Decoupling lets the order path finish at once while the notification service retries with backoff.

1. ATo avoid taxes
2. BSo the customer pays the price they saw, and stale quotes are recalculatedCorrect answer
3. CTo hide the fee
4. DTo charge surge forever

Why: Honouring a recent quote keeps the price consistent; the expiry stops a very old quote being used at a different demand level.

1. APayments
2. BOrder status updates
3. CRider assignment
4. DExpensive personalised recommendationsCorrect answer

Why: Load shedding drops nice-to-have features to protect the core path of ordering, paying and delivering.

1. AAn in-between state such as 'Waiting for the restaurant to accept'Correct answer
2. BThat the order is delivered
3. CAn error
4. DNothing until delivery

Why: Services are eventually consistent, so the UI shows honest intermediate states rather than pretending the order is final.

1. AThe rider asks for it
2. BThe restaurants are in different cities
3. CThe customers share a name
4. DBoth promised ETAs and food-quality limits still holdCorrect answer

Why: Batching saves riders, but only if neither customer's delivery is delayed beyond the promise.

1. AA single manual order
2. BLoad-testing the CDN only
3. CChaos tests that delay webhooks and kill services while checking nothing is lost or charged twiceCorrect answer
4. DUnit tests of the logo

Why: Injected failures show that timeouts, retries and compensations leave every order in a correct final state.