Silent Failure Rate, measured: 9,287 monitored automation runs across Zapier, Make, n8n and Pipedream

Across 9,287 monitored automation runs (July 1–17 2026), we recorded zero silent failures on n8n (0%, 95% CI 0–0.06%, n=6,589), Make (0%, 95% CI 0–0.94%, n=405) and Zapier (0%, 95% CI 0–0.61%, n=631) - but the moment Pipedream's free-tier quota ran out, its webhooks kept answering "success" while silently dropping 14 of 14 deliveries.

A silent failure is the automation failure that hurts most: the platform told you it worked, and it didn't. This page is a continuously running measurement of that rate - not a review, not an opinion round-up. Every number below comes from runs we fired ourselves, with both endpoints under our own control, reconciled one by one.

Scoreboard (webhook-triggered workflows)

Latency = fired-at → receipt-at-our-receiver, identical network path for every platform, so the numbers are comparable with each other (not with a vendor's internal benchmark). Median delivery: n8n 935 ms, Make 1.01 s, Zapier 4.32 s - a 5× spread on the same workload.

Beyond delivered/missed, we also check partial executions (a 2-action or 2-branch run that only half-completed: 0 observed), duplicates (0 observed) and filter leaks (runs a filter should have stopped but didn't: 0 observed). All zero so far.

The one real failure mode we caught: the quota wall

On July 2 at 14:24 UTC, our Pipedream free-tier credits ran out mid-measurement. From one event to the next - five seconds apart - delivery went from 100% to 0%. The part that matters: Pipedream's webhooks continued to return a success response for every event it then silently discarded (14 of 14 output-expected runs, 95% CI 78–100%). The sender gets no error, no queue, no replay - just an "ok" and a black hole. We re-verified the behaviour 19 hours later, past the documented daily reset time: still accepting, still dropping.

We do not count these runs in Pipedream's SFR - they are a billing-edge behaviour, not an engine failure (before the wall, Pipedream ran 76 runs without a single drop). But if you run production workloads on a metered free tier, this is the failure semantics you are signing up for: at the quota boundary, "accepted" stops meaning "delivered".

The ledger's first "miss": a forensic - and why it is not a silent failure

On July 7 at 04:13 UTC the reconciler flagged its first-ever missed n8n run: one event in a 10-event burst, expected 2 receipts, got 0. We treat a first miss as an incident, so here is the full trace, from three independent logs:

• The controller's log shows the webhook POST for that one event failed at send time - n8n never returned success. The events fired 600 ms before and after it were accepted and delivered normally.
• n8n's own execution log has no record of the event at all - 9 executions for 10 fires, all successful. The event was refused at the door, not accepted and dropped.
• The latency ledger shows why: our n8n runs on a deliberately minimal free-tier 1 GB VM, and the burst briefly saturated it. One in-flight run took 10.2 s against a ~1 s median, webhook responses stalled for ~8 s, one POST was refused, and within a minute everything was back to normal.

Classification: a loud failure, not a silent one. The sender saw the error at delivery time - exactly the signal a silent failure never gives you - so this run is recorded as rejected_at_send in the raw CSV and excluded from SFR's numerator and denominator, the same rule applied to every platform. n8n's zero above means: of 6,589 runs n8n accepted, none vanished.

Two honest takeaways. First, the $0 self-hosted option has real limits: a tiny VM can stall under a burst, and if your sender doesn't check the webhook response, a loud failure quietly becomes your silent one - retry on non-2xx. Second, note whose pick this incident dings: n8n tops our own recommendations, and this report stays up anyway. That is the fairness rule working as intended.

Edge probe: we took our endpoint down for 30 minutes and measured what each platform did

A reader of our data asked the right question: fine, nothing fails under normal operation - what happens when the destination goes down? So we measured it (August 1, 2026). Our receiver answered 503 to experiment-tagged events for a 30-minute window; we fired 5 webhook events at each platform during the outage, all platform settings at their defaults, and then watched for 24 hours. These runs are excluded from the SFR scoreboard above - stress experiments must never share a denominator with normal operation.

* One POST refused at the door by the deliberately small VM - a loud failure the sender saw. Retry features that exist but ship disabled: Zapier Autoreplay, n8n per-node Retry on Fail; a default n8n install also notifies nobody unless you configure an Error Trigger workflow.

The observability trap we fell into ourselves. During Make's 60-minute backoff wait, we checked every surface Make offers: the webhook queue showed 0 items, the incomplete-executions list (the URL calls it dlq) was empty, and the history showed no activity for nearly an hour. Every indicator said the events were gone - we wrote them off in our notes. Seven minutes later Make delivered all five. Events in the retry-wait state are invisible in Make's UI: an operator checking mid-incident will reasonably conclude the data is lost, and may re-send by hand - earning a duplicate when the retry engine wakes up. The retries work; you just can't see them pending.

This experiment now has its own page with the full retry timelines, per-platform playbook advice and FAQ.

Honest summary: every accepted event was ultimately recoverable on all three platforms - none of this is silent data loss. The differences are who does the work, when, and at what price: Make heals itself inside ~80 minutes (for outages inside its backoff window) and charges you per attempt; Zapier and n8n preserve failed runs indefinitely but wait for a human, and only Zapier tells you promptly that anything happened. Whichever you run: make the sender check response codes, and know which of these three behaviours your incident playbook assumes.

Scheduled (polling) workflows

A separate always-on workflow polls our data source every 30 minutes on each platform, and we track whether every scheduled tick actually happened and whether every new item was picked up.

*Make's one missed item (July 4) changed during a ~9-hour window in which the scenario sat deactivated after we edited it - Make leaves an edited scenario switched off until you re-enable it on the scenario page, and the next item superseded this one before we noticed. Our misconfiguration, logged as a harness-side incident, not a Make miss (the gotcha itself is worth knowing about). While active, Make has delivered every item and every scheduled poll. A cost observation previewed here (full cost benchmark coming): on Make, every empty poll of this workflow consumes 2 billable operations by design (poll + state lookup); on self-hosted n8n the same empty poll costs $0. Polling-heavy workloads pay a standing tax on per-operation platforms even when nothing happens.

Method - why these numbers are comparable

• Identical workflows everywhere. Four canonical flows (single action; filter + two actions; 30-minute poll; two parallel branches) rebuilt step-for-step on every platform - same trigger type, same step count, same HTTP calls.
• Both endpoints are ours. Events enter via each platform's webhook and exit as an HTTP POST to our own receiver. No third-party connectors - a Gmail outage can't masquerade as a platform failure.
• Every run is ID-tagged and reconciled. The controller writes a ledger entry before it fires and records whether the webhook accepted the POST; a reconciler classifies each run as delivered / missed / partial / duplicate / filtered / rejected-at-send. Rejections are loud failures the sender sees - they are reported, but never blended into SFR. No sampling of logs - full census.
• Plans measured: Zapier Professional and Make's paid plan (both paid for by us - nobody gives us free accounts), n8n self-hosted on a 1 GB cloud VM, Pipedream free tier. Sampling rates differ by billing model: per-task platforms get smaller samples (hence wider intervals, honestly labeled); self-hosted n8n carries the bulk sample.
• Window: continuous since July 1, 2026. Editor-mode test runs are excluded; the full per-run ledger is downloadable below.

Limitations we know about: paid-platform samples are still small (their intervals say so); latency includes our receiver's network hop (identical for all platforms); results describe these specific plans in this specific window, and the meter keeps running - numbers tighten every week.

Reading 0% honestly

Every platform above currently shows zero silent failures - and those zeros are not equal. Zero in 631 runs still allows a true rate near 0.61%; zero in 6,589 runs pins it below 0.06%. That is why this page reports Wilson 95% confidence intervals and keeps accumulating: rare failures only become visible in large samples. If a vendor quotes you a reliability number without a sample size, they are quoting a feeling.

FAQ

Raw data

Full per-run ledger (run ID, platform, workflow, fired-at, expected vs received, outcome, per-receipt latency): benchtruth-runs.csv · CC BY 4.0 · cite as "BenchTruth reliability dataset, benchtruth.com/reliability".