Webhook Retry Handling

GIFQ delivers webhooks at least once. We retry a delivery that fails in transit or comes back 5xx, on a widening schedule. We abandon a delivery your endpoint rejects with a 4xx.

This page is the contract for that behaviour. Read Overview first, since signature verification and the two body shapes apply to every retry.

What gets retried

How the delivery failed decides whether we retry, not which event it was.

Backoff schedule

Up to 8 attempts, meaning the first delivery plus 7 retries. The wait grows polynomially, and we add up to 15% random jitter so that many failing deliveries do not synchronise.

An outage shorter than about an hour usually gets absorbed without loss. An outage longer than roughly 90 minutes will drop events permanently.

When an event is abandoned

An event stops being retried when any of these is true.

• Your endpoint returned 2xx, so it is delivered and nothing further is sent.
• Your endpoint returned 4xx or 3xx, so we mark it permanently failed on that attempt.
• All 8 attempts went by without a 2xx, so we mark it permanently failed.

Rate limiting does not consume attempts

We cap deliveries at 5 per second per account. Anything over that gets delayed and rescheduled. It is not an attempt, not a failure, and does not count against the 8.

So a burst, such as an order with 500 recipients finishing at once, spreads over seconds or minutes rather than getting dropped. It also means arrival order is not delivery order.

Late arrivals beyond the backoff window

An event can arrive much later than the 90 minute window suggests. We record deliveries durably before dispatching them, and a sweeper re-queues anything that was never actually sent, for example when a dispatch was lost to a process restart. That sweep runs every 15 minutes and respects the same 8-attempt ceiling.

Design for this. An event reporting a state change from hours ago is normal, so do not reject it on age alone.

Deduplicating retries

At-least-once delivery means your handler will see the same event twice. Deduplication is your job, and the wrong key silently does nothing.

Build the key from the body instead. The event name, the identity of the record that changed, and updated_at stay stable across retries, because a retry replays the byte-identical body we captured when the event fired.

# Rails. Dedupe on a body-derived key, not on the delivery header.
def idempotency_key(body)
  case body['payout_type']
  when 'gift_cards'
    [body['event'], body['subject_type'], body['recipient_uuid'], body['updated_at']].join(':')
  when 'crypto'
    [body['event'], body.dig('data', 'id')].join(':')
  end
end

Let the insert of that key enforce uniqueness, with a unique index and a rescued conflict, rather than a read-then-write check. Two retries of the same event can land on two of your workers at once.

Ordering

Retries break ordering by construction. A payout.processing we retry for 40 minutes can land well after the payout.fulfilled that followed it.

Sort on updated_at from the body and discard transitions older than the state you already hold. Never infer order from arrival time, and never assume a terminal event is the last thing you will receive.

What your endpoint should do

1. Read the raw body and verify X-Gifq-Signature. Reject with 401 on mismatch, since a forged body is one you genuinely never want replayed.
2. Compute the idempotency key. If you have seen it, return 2xx straight away.
3. Store the body durably. Do not process it inline.
4. Return 2xx inside the 10 second read timeout, ideally in well under a second.
5. Do the real work asynchronously, off the request.

Troubleshooting