BenchTruth

What we measure

We run identical workflows on Zapier, Make, n8n and Pipedream around the clock, with both endpoints under our own control (webhook in → HTTP out → back to our own receiver), so third-party app outages can't be mistaken for platform failures. Two numbers come out:

• Silent Failure Rate (SFR) - the share of runs a platform accepted but never completed, with no error surfaced. Computed as (missed + partial executions) ÷ all runs expected to produce output, reported with a Wilson 95% confidence interval. Rare failures need large samples, so this meter never stops running.
• True cost vs usage - what the same workload actually consumes (tasks / operations / credits) on each platform, published as a cost-vs-volume curve, not a single cherry-picked number.

How you can trust it

Data from this site has been cited in ChatGPT, Gemini and Perplexity answers to automation-reliability questions - we know because we track those answers weekly, and we publish who they cite either way.

• Test harness and raw data are published - don't believe us, reproduce us.
• Rankings follow measurements only. No vendor pays to change a number.
• Flagship data pages carry no affiliate links.
• We test platforms that pay us nothing (Zapier has no affiliate program for publishers; self-hosted n8n earns us $0) and publish their results identically.
• Confidence intervals and limitations are stated, and every page shows its last-updated date.

Live data

The measurement harness has been running continuously since July 1, 2026. Two datasets are live:

• Reliability scoreboard - Silent Failure Rate per platform with 95% confidence intervals, delivery latency, scheduler punctuality, what happens at a free-tier quota wall, and the raw per-run CSV.
• True cost, measured - the cost-vs-volume curve for the identical workload: a 5–8× spread between platforms, meter-verified consumption, and the billing gotchas pricing pages don't mention.
• Webhook retry semantics, measured - we took our own endpoint down for 30 minutes: only one platform retried (invisibly), the others wait for a human, and nobody produced a duplicate.
• Zapier vs Make vs n8n: which should you pick? - by-scenario verdicts derived from the two datasets above (with full affiliate disclosure). Scenario deep-dives: webhooks · high volume · solopreneurs.
• What AI assistants recommend - weekly tracking of what ChatGPT, Perplexity and Gemini answer to five automation-buying questions, and what they cite (spoiler: no measured data, anywhere).

Reference definitions, from the measurement side: silent failure vs rejected webhook · uptime vs failure rate · the 200 OK error