The 10-Second Budget: Why Webhooks Fail Before Your Code Even Runs
The 10-Second Budget: Why Webhooks Fail Before Your Code Even Runs
When developers build event-driven systems, they often make a quiet assumption: "If my code processes the payload in under a couple of seconds, my webhook endpoint will succeed."
In production, that assumption falls apart. Webhooks fail silently, trigger cascading retries, or get disabled by the provider entirely — even when your application code runs fast.
The reason is that your code never gets a clean slate. It's operating inside a strict response window enforced by the provider — Slack, Shopify, GitHub, Stripe, Twilio, and nearly everyone else. Whether it's a 10-second ceiling or a brutal 3-second one, this window is a hard boundary. By the time your route handler receives the payload, a hidden waterfall of network hops, cryptographic checks, and runtime startup has already burned through a meaningful chunk of your budget.
This piece breaks down the anatomy of a webhook request, the real, current timeout numbers for major providers, where hidden latency actually accumulates in 2026, and why decoupling ingestion from processing is still the fix — using real, verifiable infrastructure rather than a made-up product name.
The Reality of Provider Timeout Windows
Third-party platforms send huge volumes of webhooks, so they enforce aggressive timeouts to protect their own infrastructure from hanging sockets. If your server doesn't return a 2xx status before the deadline, the provider marks the delivery failed.
Here's what's actually documented today, not the vaguely-remembered numbers that circulate in blog posts:
| Provider | Response Window | Retry Behavior | What Happens on Persistent Failure |
|---|---|---|---|
| Slack | 3 seconds (hard, non-configurable) | Up to 3 automatic retries; with "Delayed Events" enabled, additional hourly retries for 24h | Retries stop; the x-slack-retry-reason header tells you why a retry fired |
| Shopify | 5 seconds total (plus a separate 1-second connection timeout) | Up to 19 retries spread over 48 hours | Endpoint is automatically disabled after sustained failures over that window |
| GitHub | 10 seconds | No automatic redelivery. A timed-out delivery is simply marked failed — you must redeliver manually or via the API | GitHub does not auto-disable webhooks for failed deliveries (it only auto-disables ones restricted by OAuth app access policies) |
| Stripe | ~20 seconds (commonly cited figure; Stripe's own guidance is simply "respond before any logic that could cause a timeout") | Up to ~16 attempts with exponential backoff over roughly 3 days | Email alerts, then the endpoint is disabled after continuous failure across that window |
| Twilio | 15 seconds for Voice/SMS request URLs; 5 seconds for Conversations webhooks | Voice/SMS request-URL webhooks are not retried — Twilio falls straight to your configured Fallback URL. Status callbacks are retried, up to 3 times with backoff | No fallback configured means the inbound event is simply dropped |
A few corrections worth calling out explicitly, because they get repeated incorrectly all over the web:
GitHub does not automatically retry failed deliveries. If your handler times out, that event is gone until you manually redeliver it via the dashboard or the REST API — GitHub explicitly recommends writing a scheduled script to poll for and redeliver failed events.
Twilio doesn't "retry" incoming message and call webhooks in the usual sense — it fails over to a separate Fallback URL immediately. Only status callbacks get real retries.
Shopify's connection timeout (1 second) is tighter than its total response timeout (5 seconds). Your server has to accept the TCP connection in under a second, which is easy to miss if you're only measuring handler execution time.
The Hidden Latency Waterfall: Where the Budget Actually Disappears
An incoming HTTP POST doesn't materialize instantly inside your route handler. Between the provider emitting the event and line 1 of your function executing, the request crosses several layers:
1. DNS Resolution ~ 20ms - 100ms
2. TCP + TLS 1.3 Handshake ~ 50ms - 250ms
3. Ingress / Load Balancer routing ~ 10ms - 50ms
4. Cold start (if serverless, see below) ~ 0ms - 3,000ms+ (highly variable)
5. Signature verification + body parsing ~ 10ms - 100ms
6. Database connection pool acquisition ~ 10ms - 1,000ms+ under load
The single biggest variable here — and the one most articles get outdated on — is cold starts.
Cold starts are much less scary than they used to be (with big caveats)
The "500ms–3,000ms" cold start figure that circulates constantly is stale for the common case. Current production benchmarks on AWS Lambda in 2026 put Node.js and Python cold starts around 200–400ms at P50 on arm64 (Graviton), and AWS reports cold starts occurring in under 1% of invocations for warm-traffic workloads. VPC-attached cold starts, which used to add 10+ seconds, were fixed back in 2019. Rust and custom runtimes can cold-start in under 20ms.
The caveat: Java and .NET are still slow to cold-start — multiple seconds without mitigation — unless you use something like AWS Lambda SnapStart, which brings Java down from roughly 5 seconds to under 200ms in AWS's own benchmarks. And cold starts still matter disproportionately for webhooks specifically, because a webhook consumer that only fires a few times an hour is exactly the profile that gets recycled between invocations, unlike a busy user-facing API that stays warm.
Practical takeaway: don't assume cold starts are dead, but also don't budget 3 seconds for them if you're running Node.js or Python on modern Lambda without a VPC. Measure your actual P95/P99 init duration instead of using a rule of thumb.
Database connection pool starvation is still very real
// Common anti-pattern
app.post('/api/webhooks/stripe', async (req, res) => {
verifySignature(req);
const dbClient = await pool.connect(); // <-- hangs here under load
const customer = await dbClient.query('SELECT * FROM users WHERE stripe_id = $1', [req.body.customer]);
// ...business logic...
res.status(200).send('OK');
});
If traffic spikes and your connection pool is exhausted, your handler blocks waiting for a socket. That wait counts against your timeout budget just as much as slow business logic does.
The Cascade of Failure
When a webhook does time out, the fallout goes well beyond one failed HTTP call:
Webhook times out
│
▼
Provider assumes delivery failed
│
▼
Provider retries (schedule varies wildly by provider — see table above)
│
┌────┴────┐
▼ ▼
Duplicate Out-of-order
processing processing
│ │
└────┬────┘
▼
Endpoint disabled after sustained failure (Slack, Shopify, Stripe)
Phantom processing. A timed-out connection doesn't stop your server from finishing the work it already started. The provider marks it failed even though your database write eventually succeeds.
Duplicates. Because the provider thinks the first attempt failed, a retry arrives later and your server processes the same event twice — double-charged cards, duplicate emails, overwritten records — unless you have real idempotency handling.
Out-of-order delivery. None of the major providers guarantee ordered delivery on retry. An order.updated event stuck in a retry queue can arrive after a later order.cancelled event succeeds on the first try.
Automatic disablement. Shopify and Stripe will both automatically disable an endpoint after it fails consistently over their respective windows (48 hours and roughly 3 days). GitHub, notably, will not — but it also won't retry for you, so silent data loss is the actual risk there instead.
The Architectural Flaw: Treating Webhooks Like a REST API
The root cause is usually structural. A normal REST client waits synchronously for a result:
Client Request → Business Logic → Database Query → HTTP 200 Response
Applying that pattern to webhooks means asking an external service to hold a TCP socket open while you parse JSON, hit three tables, generate a PDF, and post to Slack. The provider doesn't care about your business logic outcome — it only cares whether you received the payload.
The Fix: Decouple Ingestion from Processing
Webhook Provider (Stripe/Shopify/etc.)
│ 1. HTTP POST
▼
┌───────────────────────────────┐
│ Ingestion layer (edge/API) │
└───────────────────────────────┘
│ │
│ 2. Push to queue │ 3. Return 2xx/202 immediately
▼ ▼
┌────────────────┐ Provider closes socket, happy
│ Durable queue │
│ (Redis/SQS/ │
│ Kafka) │
└────────────────┘
│ 4. Async pull
▼
┌───────────────────────────────┐
│ Background worker layer │
│ (DB writes, 3rd-party calls, │
│ PDF generation, etc.) │
└───────────────────────────────┘
The golden rule: acknowledge receipt immediately with a 2xx/202, then process asynchronously.
Building a Low-Latency Ingestion Buffer
There are two realistic approaches: run your own queue, or use a managed webhook ingestion gateway.
Approach 1: Self-hosted queue (Redis + BullMQ, or SQS)
// Node.js / Express — ingestion-only handler
import express from 'express';
import { Queue } from 'bullmq';
const app = express();
const webhookQueue = new Queue('webhook-processing', { connection: redisConfig });
app.post('/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
const sig = req.headers['stripe-signature'];
try {
if (!verifyStripeSignature(req.body, sig, process.env.STRIPE_SECRET)) {
return res.status(400).send('Invalid Signature');
}
} catch {
return res.status(400).send('Signature Verification Failed');
}
// No DB calls here — just push to the queue
await webhookQueue.add('stripe-event', {
rawPayload: req.body.toString(),
receivedAt: Date.now(),
}, {
attempts: 5,
backoff: { type: 'exponential', delay: 1000 },
});
return res.status(202).json({ received: true }); // Typically 10-40ms
});
This works well and keeps you in full control, but you own the operational overhead: queue infrastructure, backpressure handling, dead-letter queues, replay tooling, and observability.
Approach 2: A managed webhook ingestion gateway
Rather than build all of that yourself, several real, currently-maintained products specialize in exactly this problem — receiving webhooks at the edge, acknowledging instantly, and feeding your backend at a rate it can handle:
Hookdeck — purpose-built for inbound webhook orchestration: durable queueing with backpressure, 100+ pre-configured provider sources, filtering, transformations, and replay. Closed-source, cloud-only.
Svix Ingest — the inbound counterpart to Svix's outbound webhook-delivery platform (Svix Dispatch, which co-authored the Standard Webhooks spec). A good fit if you're already using Svix to send webhooks to your own customers, or need HIPAA/PCI/CCPA-ready infrastructure out of the box.
Convoy — a fully open-source webhooks gateway that handles both inbound and outbound delivery, useful if self-hosting is a hard requirement.
Hook0 — open-source (SSPL-1.0), EU-hosted, GDPR-oriented; more focused on outbound webhook delivery to your own customers than inbound ingestion.
AWS EventBridge Pipes / API destinations, Upstash QStash, or plain SQS — cloud-native, lower-level building blocks if you want managed queueing without a dedicated webhooks product.
The pattern is the same regardless of vendor: a globally-distributed ingress layer does signature verification and returns a 2xx in single-digit-to-low-double-digit milliseconds, then streams the payload to your backend asynchronously, with retries and dead-letter handling built in. Which one fits depends on whether you mainly need to receive webhooks reliably (Hookdeck, Svix Ingest), send webhooks to your own customers (Svix Dispatch, Hook0), or want to self-host end to end (Convoy, Hook0).
Best Practices Checklist
1. Separate routing from execution. Don't query your primary database inside the webhook route. Validate the signature, push to a queue, return 202.
2. Optimize network ingress. Host close to the provider's origin, or route through an edge layer with HTTP/2 and TLS session resumption to avoid redundant handshakes.
3. Handle cold starts realistically, not by rule of thumb. On modern Lambda (Node.js/Python, arm64, no VPC), cold starts are typically sub-500ms — measure yours instead of assuming worst-case numbers from 2019-era benchmarks. If you're on Java or .NET, use SnapStart or provisioned concurrency.
4. Implement real idempotency, keyed off the provider's own event ID:
Stripe: the event's
id(e.g.evt_1N...)GitHub: the
X-GitHub-DeliveryheaderShopify: the
X-Shopify-Webhook-IdheaderTwilio: the
I-Twilio-Idempotency-Tokenheader (on supported products)
Store processed IDs in a fast cache with a TTL of 24–72 hours to drop duplicate retries before they touch your business logic.
5. Monitor P95/P99, not averages. A 3-second average response time can hide a P99 that's blowing past Slack's 3-second hard limit on a meaningful slice of traffic.
Before vs. After
❌ Synchronous processing (timeout-prone):
@app.post("/webhooks/shopify")
async def handle_shopify_webhook(request: Request):
payload = await request.json()
conn = psycopg2.connect(DATABASE_URL)
cursor = conn.cursor()
cursor.execute("SELECT * FROM inventory WHERE product_id = %s", (payload['product_id'],))
item = cursor.fetchone()
time.sleep(4.5) # simulating real business logic — risks Shopify's 5s window
cursor.execute("UPDATE inventory SET stock = stock - 1 WHERE product_id = %s", (payload['product_id'],))
conn.commit()
return {"status": "success"}
✅ Decoupled async ingestion:
@app.post("/webhooks/shopify")
async def handle_shopify_webhook(request: Request):
raw_body = await request.body()
hmac_header = request.headers.get("X-Shopify-Hmac-SHA256")
if not verify_shopify_hmac(raw_body, hmac_header):
return Response(status_code=status.HTTP_401_UNAUTHORIZED)
event_data = {
"event_id": request.headers.get("X-Shopify-Webhook-Id"),
"payload": raw_body.decode("utf-8"),
}
r.lpush("shopify_webhook_queue", json.dumps(event_data))
return Response(status_code=status.HTTP_202_ACCEPTED) # ~10-15ms
Conclusion
The response window webhook providers give you isn't generous — it's a safety ceiling, and the numbers above are the real, currently-documented ones, not the rounded-off figures that get copy-pasted across the internet. Once you account for network latency, TLS handshakes, occasional cold starts, and database contention, your business logic rarely gets the full window.
The fix hasn't changed even as the numbers have: acknowledge immediately, process asynchronously, and put a real queue — self-hosted or managed — between the provider and your business logic.