Technology handbook
Webhook
Receiving, verifying, acknowledging, retrying and replaying inbound events safely.
- Topics
- 31
- Sections
- 5
Receiver pipeline
Raw body capture
Capture the unparsed body before any middleware reads it, or signature verification cannot pass.
- Read the request body as exact bytes before JSON parsing.
- Prevent middleware from rewriting the signed payload.
- Bound body size before allocating large buffers.
Signature verification step
Verify before parsing, and treat a failed verification as an unauthenticated request.
- Check the sender's signature using its documented verification scheme.
- Reject unauthenticated payloads before acting on their contents.
- Use a maintained verifier when the provider supplies one.
Idempotency check
Look up the event id and stop early when this delivery has already been processed.
- Use an atomic event-ID claim rather than a check-then-insert race.
- Distinguish processing, completed and retryable-failure states.
- Retain deduplication records for the sender's replay window.
Acknowledge-then-work ordering
Record the event durably, answer 2xx, and only then run the handler that may be slow.
- Persist the verified event before returning a success response.
- Let a worker perform slow or retryable business processing.
- Do not acknowledge an event that would be lost on process exit.
Queue handoff
Hand the event to a queue so a slow handler cannot cause a delivery timeout.
- Make event persistence and queue publication recoverable together.
- Use an outbox when separate writes can fail between systems.
- Pass event references instead of oversized queue messages.
Failure classification
Distinguish a retryable failure from a permanently bad payload before choosing a status code.
- Identify invalid signatures and schemas as permanent request problems.
- Treat temporary storage or queue failures as recoverable.
- Follow the sender's documented retry interpretation of status codes.
Dead-letter handling
Park an event that cannot be processed where a person will actually see it.
- Store exhausted events with the reason and attempt history.
- Assign an owner to inspect and repair failed deliveries.
- Reprocess through the same idempotent business handler.
- Capture the last error and relevant correlation identifiers.
- Alert on growing dead-letter age as well as count.
Replay endpoint
Re-process a stored event on demand, using the same handler the live delivery used.
- Authorize operators before allowing a stored event to be replayed.
- Record who requested the replay and why.
- Preserve the original event identity to prevent duplicate effects.
Receiving
Endpoint design
Expose a dedicated route that does one thing and returns quickly.
- Expose a dedicated HTTPS route for the integration.
- Keep request processing short and bounded.
- Separate webhook authentication from normal browser session flows.
Raw body access
Read the unparsed body, since signature verification depends on exact bytes.
- Use the provider's exact payload bytes for signature calculation.
- Read the body once and reuse the verified buffer.
- Parse into a domain object only after authenticity checks.
Status code semantics
Return 2xx to acknowledge and a retryable status only when a retry would help.
- A 2xx response usually tells the provider delivery succeeded.
- Non-success responses may trigger retries under provider-specific rules.
- Avoid returning success before the event is durably accepted.
Fast acknowledgement
Accept the event, then process it outside the request.
- Acknowledge after durable acceptance rather than business completion.
- Keep response latency below the provider's timeout.
- Monitor worker failures separately from delivery acknowledgement.
Payload validation
Parse the body against a schema before trusting any field.
- Validate event type, required fields and expected value formats.
- Handle unknown event types without unsafe fallback behavior.
- Separate schema validity from permission to change a resource.
Content types and encoding
Handle the encoding the sender actually uses.
- Check the declared content type against supported formats.
- Preserve the byte representation used by the signature scheme.
- Reject unsupported encodings and oversized decoded payloads.
Verifying
Signature verification
Recompute the HMAC over the raw body and compare in constant time.
- Bind verification to the provider's expected signed message format.
- Compare computed signatures with a constant-time method.
- Validate the signing secret belongs to the intended endpoint.
Timestamp tolerance
Reject old requests so a captured payload cannot be replayed later.
- Check signed timestamps against an explicit tolerance window.
- Keep receiver clocks synchronized to avoid false rejections.
- Never trust an unsigned timestamp as replay evidence.
Secret rotation
Accept two secrets during a rotation window without downtime.
- Allow a controlled overlap while replacing signing secrets.
- Record which configured secret verified the delivery.
- Retire old secrets after the provider's transition period.
Replay protection
Record seen event ids and refuse a repeat of a one-time action.
- Combine signature verification with timestamp and event-ID checks.
- Retain seen IDs long enough to cover expected redelivery.
- Ensure concurrent duplicates cannot both claim the same operation.
IP allow lists
Use network restrictions as defence in depth, never as the only check.
- Treat source-IP restrictions as an additional control.
- Use the provider's maintained ranges and trusted proxy configuration.
- Keep cryptographic verification even when the address is allowed.
mTLS and shared tokens
Authenticate the sender where signatures are unavailable.
- Use mTLS when both sender and receiver support client certificates.
- Rotate shared tokens and send them only over HTTPS.
- Avoid placing authentication tokens in logged query strings.
Processing reliably
Idempotency keys
Key the effect on the event id so repeats converge on one outcome.
- Scope the deduplication key to provider, account and event.
- Enforce uniqueness atomically in durable storage.
- Reuse the same key when invoking an idempotent downstream API.
Queueing the work
Hand off to a durable queue so the handler stays fast.
- Enqueue validated events with bounded retry and retention settings.
- Run business side effects in a separately monitored worker.
- Use acknowledgements only after the worker's durable result.
Ordering assumptions
Design for events arriving out of order, because they will.
- Assume deliveries can arrive late or out of sequence.
- Compare resource versions before applying stale events.
- Fetch authoritative current state when ordering is ambiguous.
Retry and backoff
Understand the sender's retry schedule and align your own.
- Use exponential backoff with jitter for transient outbound failures.
- Honor Retry-After where the destination defines it.
- Stop retrying when the attempt or age budget is exhausted.
Replay and backfill
Reprocess a range of events safely after a bug is fixed.
- Backfill missing events from an authoritative source when available.
- Apply the same validation and deduplication rules during replay.
- Rate-limit recovery traffic so normal processing remains healthy.
Operating
Logging and correlation
Record the event id end to end so one delivery is traceable.
- Correlate provider event ID, delivery attempt and internal job ID.
- Log verification and processing outcomes without secret payload fields.
- Preserve enough metadata to trace one event end to end.
Monitoring delivery
Alert on failure rate and on a sudden absence of events.
- Track acknowledgement latency, failure rate and retry volume.
- Monitor backlog age and time to completed business processing.
- Alert when accepted events stop making progress.
Local development
Receive real events on a development machine through a tunnel.
- Use the provider's supported local forwarding or test tooling.
- Keep development endpoints and signing secrets separate.
- Exercise duplicate deliveries and invalid signatures locally.
Testing handlers
Test signature rejection and duplicate delivery, not just the happy path.
- Test authentic, malformed, duplicate and reordered event fixtures.
- Verify persistence happens before a success acknowledgement.
- Simulate worker failures and confirm retry-safe side effects.
Versioning
Handle a sender's payload version changing under you.
- Track provider API and payload versions explicitly.
- Support a migration overlap when old and new shapes coexist.
- Reject or quarantine incompatible events with clear diagnostics.
Outbound webhooks
Send your own events with signatures, retries and a documented contract.
- Sign the exact body sent to each subscribed destination.
- Retry delivery under a documented bounded policy.
- Validate destination URLs and prevent server-side request forgery.
Sources
- Webhook official documentation