Monitor Stripe webhooks: know when events stop arriving or go missing
Monitor Stripe webhooks with a heartbeat. Your webhook handler pings a unique URL after every event it processes, and you get an alert when no ping arrives within your longest normal gap. Add a nightly job that compares Stripe with your database and sends the mismatch count as a value, so anything above 0 alerts.
What can go wrong with Stripe webhooks
- The handler starts returning errors. A deploy breaks the route, a rotated signing secret makes every signature check fail, or a firewall rule blocks Stripe's servers. Payments still succeed, so nothing looks wrong on your side. Stripe retries a failed event for up to three days in live mode, then stops.
- Events arrive but nothing happens. The handler answers 200 first and fails later, swallows an exception, or ignores an event type after a change. Stripe sees a success, so there is no failed delivery to find.
- No events at all. The endpoint was deleted in the Dashboard, a new environment was never registered, or the URL changed in a migration. Your logs are empty, which looks the same as a quiet day.
- Duplicates and out-of-order events. Stripe can send the same event more than once and does not promise the order. A handler that is not idempotent credits twice or misses an update.
The signals that matter, with alert thresholds
| Signal | Alert when | Why it matters |
|---|---|---|
| Heartbeat from the webhook handler | no ping within the longest normal gap plus grace | Events stopped arriving, or the handler stopped processing them. |
| Nightly reconciliation mismatch count (<code>?value=</code>) | above 0 | Stripe and your database disagree: an event was lost, failed or handled twice. |
| Failed deliveries for your endpoint in Stripe | any failed event | Stripe is retrying because your handler returned an error or timed out. |
| Webhook URL checked from outside | connection error, TLS error or an unexpected status | Stripe cannot reach your server at all. |
Thresholds are starting points. Tune them after a week of data: an alert nobody acts on should be raised or removed.
How to check Stripe webhooks by hand
curl -sS -G https://api.stripe.com/v1/events -u "$STRIPE_KEY:" \
-d delivery_success=false -d limit=10 | jq '.data[] | {id, type, created}'
stripe trigger checkout.session.completed
stripe listen --forward-to localhost:8000/webhooks/stripe
Run these with a test mode key first. delivery_success=false lists events where at least one webhook delivery failed.
1. Ping a heartbeat on every processed event
Create a Vigil Heartbeat and call its URL at the end of your webhook handler, after the event was verified and your own work is done. Ping after processing, not before: a ping at the top of the handler proves only that a request arrived. Use a short timeout and ignore errors, so monitoring can never slow down or break the webhook.
import urllib.request
HEARTBEAT_URL = "https://puzaricloud.in/ping/<token>"
def ping_webhook_heartbeat():
"""Call at the end of the handler, after the event was verified and processed."""
try:
urllib.request.urlopen(HEARTBEAT_URL, timeout=5)
except Exception:
pass # never let monitoring break the webhook itself
@app.post("/webhooks/stripe")
async def stripe_webhook(request: Request):
event = stripe.Webhook.construct_event(await request.body(), request.headers["stripe-signature"], WEBHOOK_SECRET)
handle_event(event) # save the order, unlock the account, ...
ping_webhook_heartbeat()
return {"received": True}// Node 18 and later: last line of the handler, after the event was verified and processed
fetch("https://puzaricloud.in/ping/<token>", { signal: AbortSignal.timeout(5000) }).catch(() => {});Every event pings, so busy hours send many pings. That is fine: the heartbeat only cares about the time since the last one.
2. Set the period to the longest normal gap
The heartbeat alerts when no ping arrives within its period plus the grace. Use the longest gap you see on a normal day. If events come in all day, a period of 3 hours is a good start. If nights are quiet, a fixed period would alert every night, so use a cron expression instead: 0 9-21 * * * in your time zone with a grace of 180 minutes expects events between 9:00 and 21:00 and leaves the night alone. Look at a week of Stripe events to pick the numbers, then tighten them.
A quiet heartbeat does not always mean a broken webhook. Sometimes sales are slow. Either way, you hear about it the same day.
3. Reconcile Stripe with your database every night
The heartbeat tells you events are arriving. It cannot tell you that every one was handled. A small nightly job can: list yesterday's paid Checkout Sessions in Stripe, look for each one in your orders table, and send the number of missing ones with the ping as ?value=. Create a second heartbeat for it with Alert if value is above set to 0. Any mismatch alerts and opens an incident; the next clean night closes it.
# reconcile_stripe.py: yesterday's paid Checkout Sessions in Stripe vs orders in your database
import time, urllib.request
import stripe
stripe.api_key = STRIPE_KEY # a restricted key with read access to Checkout Sessions is enough
since = int(time.time()) - 26 * 3600 # a little overlap so nothing falls between two runs
in_stripe = {s.id for s in stripe.checkout.Session.list(created={"gte": since}, status="complete", limit=100).auto_paging_iter()
if s.payment_status == "paid"}
in_db = set(db.scalars("SELECT stripe_session_id FROM orders WHERE created_at >= now() - interval '26 hours'"))
mismatches = len(in_stripe - in_db) # paid in Stripe, missing here: a webhook was lost or failed
urllib.request.urlopen(f"https://puzaricloud.in/ping/<token>?value={mismatches}", timeout=10)# every night at 02:30: the job pings with ?value=<mismatches>; the heartbeat alerts above 0 30 2 * * * /usr/bin/python3 /opt/app/reconcile_stripe.py || curl -fsS -m 10 https://puzaricloud.in/ping/<token>/fail
If the job itself crashes, the /fail ping alerts at once, and if it never runs, the heartbeat goes late. The same pattern works for subscriptions, refunds or payouts: count what does not match and alert above 0. More on values in the cron job guide.
4. Check Stripe's failed deliveries
In the Stripe Dashboard, open your webhook endpoint (under Developers, then Webhooks) and look at the failed deliveries. Each one shows the status code and response body your server sent, which is usually enough to find the bug. Fix the handler, then resend the failed events from the same page, or let Stripe's retries deliver them. Run the reconciliation job by hand afterwards to confirm the count is back to 0.
Make the handler safe to run twice before you resend anything: store the event ID and skip events you have already processed.
How to watch Stripe webhooks with Vigil
- In New monitor create a Heartbeat named "Stripe webhook". Use a period of your longest normal gap, or a cron expression such as
0 9-21 * * *with a grace of 180 minutes. Call its ping URL, https://puzaricloud.in/ping/<token>, at the end of the webhook handler. - Create a second Heartbeat named "Stripe reconciliation" with the schedule
30 2 * * *, a grace of 30 minutes and Alert if value is above set to 0. The nightly job pingshttps://puzaricloud.in/ping/<token>?value=$MISMATCHES. - Add an API check on your webhook URL that expects the status a plain GET returns (often 405), so you also know the endpoint is reachable from outside.
- Send the alerts somewhere you read every day: push, Slack, Teams, Telegram, Discord or email. The hosted service does not send SMS or make phone calls.
Every alert goes to the channels you pick: email, Slack, Telegram, Discord, Microsoft Teams, Pushover, webhooks or on-call paging. Failures are confirmed by a second check before anyone is woken up.
Stripe webhooks monitoring FAQ
Should the handler ping before or after processing the event?
After. A ping at the start only proves that a request arrived. A ping at the end proves the event was verified and handled, which is what you care about.
What period should the webhook heartbeat have?
The longest gap between events on a normal day, plus a grace period. Steady traffic: about 3 hours. Quiet nights: use a cron expression that only expects events during the day.
Why reconcile if I already have a heartbeat?
The heartbeat shows that events keep arriving. The reconciliation shows that each paid order in Stripe exists in your database. One lost event among hundreds keeps the heartbeat green but makes the mismatch count 1, which alerts.
Does Stripe not tell me about failed webhooks?
Stripe retries failed events for up to three days in live mode and shows them as failed deliveries in the Dashboard. A handler that returns 200 but does nothing never shows up there, and a retry window can run out over a long weekend.