How this benchmark has changed, month by month
Generated from the run ledger · 11,399 monitored runs to date · regenerated 2026-09-03 · no affiliate links on this page
Every other page here tells you what the numbers are now. This one tells you how they got there: how much the sample grew each month, how much tighter the confidence intervals became as a result, which platforms joined and which lapsed, and what each experiment found when it was published.
It matters for a boring but important reason. A reliability figure without a history is a claim; a reliability figure you can watch tighten over months is a measurement. The counts below are computed from the same ledger that produces every other number on this site, so this page cannot drift away from the data even if we forget to update it.
September 2026
+309 monitored runs this month · 11,399 cumulative.
- Zapier's plan lapsed to Free on September 3; the harness now runs at $0/month on self-hosted n8n plus free tiers.
- Latency percentiles published for the first time (p50 through p99).
| Platform | Runs this month | Cumulative | Silent-failure rate below (95%) |
|---|---|---|---|
| n8n (self-hosted) | 270 | 9,736 | 0.05% |
| Zapier | 32 | 968 | 0.46% |
| Make | paused | 540 | 0.94% |
| Pipedream | paused | 113 | 24.43% |
| Pabbly Connect | 7 | 42 | 9.18% |
August 2026
+4,049 monitored runs this month · 11,090 cumulative.
- Outage experiment (EP2): we took our own receiver down for 30 minutes and measured what each platform does. Make retried for 79 minutes and self-healed; Zapier and n8n waited for a human.
- Malformed-response experiment (EP3): platforms record a 200 OK carrying an error as a full success, and catching it costs nothing standing.
- Make's paid window ended; Pabbly Connect joined as the fifth platform.
- Reference pages published for the vocabulary this project introduced (silent failure vs rejected webhook, uptime vs failure rate, the 200 OK error).
| Platform | Runs this month | Cumulative | Silent-failure rate below (95%) |
|---|---|---|---|
| n8n (self-hosted) | 3,510 | 9,466 | 0.05% |
| Zapier | 468 | 936 | 0.47% |
| Make | 36 | 540 | 0.94% |
| Pipedream | paused | 113 | 24.43% |
| Pabbly Connect | 35 | 35 | 10.43% |
July 2026
+7,041 monitored runs this month · 7,041 cumulative.
- Harness went live (July 1) and the first measurements began accumulating.
- Zapier and Make paid windows opened; billing models verified against both meters (Zapier's task meter matched our executed-action count exactly, 45 = 45).
- Pipedream's free-tier quota ran out mid-measurement and its webhooks kept returning success while discarding every event - our first measured failure mode.
- First public data post; the benchmark began appearing in AI assistant answers five days later.
| Platform | Runs this month | Cumulative | Silent-failure rate below (95%) |
|---|---|---|---|
| n8n (self-hosted) | 5,956 | 5,956 | 0.08% |
| Make | 504 | 504 | 1.01% |
| Zapier | 468 | 468 | 0.94% |
| Pipedream | 113 | 113 | 24.43% |
Reading this table
- The last column is the point. It is the upper bound of the 95% confidence interval on the true silent-failure rate, given everything measured up to the end of that month. Every platform has observed zero silent failures throughout; what changes is how confidently we can say the true rate is small. Watching that bound fall month over month is what accumulating evidence actually looks like.
- "Paused" is not a gap in the data. Platforms stop when a paid window ends or a free tier is exhausted; their accumulated totals stay valid for the period they ran, and we say when that period was rather than quietly blending it into a current figure.
- Sample sizes are deliberately unequal. The self-hosted platform costs nothing per run so it carries the bulk sample; metered platforms get small ones. That is why their intervals stay wider, and why we publish intervals instead of bare percentages.
Current figures live on the reliability scoreboard, cost on the cost page, and delivery speed on the latency benchmark. Raw per-run data: benchtruth-runs.csv (CC BY 4.0).