← BenchTruth

Silent failure in automation platforms, Q3 2026

Quarterly report · July 1 to September 30, 2026 · 14,472 monitored runs · published 2026-11-13 · no affiliate links on this page

A quarter of measurement on five automation platforms: four hosted (Zapier, Make, Pabbly Connect, Pipedream) and one self-hosted (n8n). 14,472 monitored runs. Every figure here comes from events we fired ourselves and reconciled against receipts, not from vendor status pages, uptime monitors or user reports.

The raw per-run ledger is published under CC BY 4.0 at /license and always will be. This report is the part that is not in the data: what the numbers do and do not support, which of them we would act on, and what we got wrong. The live, continuously updated figures are on /reliability; this page is the quarter, frozen.

The one-sentence version

No platform silently dropped a single run under normal operation all quarter. Every loss we measured happened at a billing edge, not inside an engine.

That sentence is worth less than it sounds, and the rest of this report is mostly about why.

The table

PlatformRunsOutput-expectedSilent failures95% CI upper boundMeasurement window
n8n (self-hosted)12,76610,63500.04%Jul 2 to Sep 30, 91 days
Zapier96884000.46%Jul 3 to Sep 2, 61 days
Make54040500.94%Jul 1 to Aug 3, 33 days
Pipedream967604.81%Jul 1 to Jul 2, 31 hours
Pabbly Connect847205.07%Aug 19 to Sep 22, 34 days

"Output-expected" is the denominator: runs the platform accepted and that should have produced output. Runs correctly stopped by a filter branch are excluded, because a filter doing its job is not a failure. Runs refused at send time are excluded from both numerator and denominator, because a refusal is loud and the sender can retry. Runs fired during an outage on our own side are excluded and listed separately (see below).

Reading those intervals

Every platform scored zero. The zeroes are not comparable and the intervals are the reason.

n8n's upper bound is 0.04%. Pabbly's is 5.07%. Those are two orders of magnitude apart, and they describe the same observed result. The difference is entirely sample size. A zero in 72 runs is consistent with a true rate of one in twenty. A zero in 10,635 is not.

So the honest reading of this table is not "all five are equally reliable". It is:

If a comparison page tells you a platform has a 0% failure rate without giving you n and an interval, it is not reporting a measurement. It is reporting an absence of evidence and hoping you read it as evidence of absence.

The windows are not equal, and that is the biggest caveat in this report

Four of the five platforms stopped before the quarter ended, every one of them for a billing reason rather than a technical one:

Only n8n ran the full quarter, and it ran it because we self-host it and nobody meters us.

This means the sample sizes in the table are a record of our budget, not of the platforms' behaviour. n8n has nearly 150 times Pabbly's sample because n8n is free to run at volume, not because it is more deserving of measurement. Anyone comparing 0.04% to 5.07% and concluding something about engineering quality has read the table backwards.

The quarter's actual finding: the failure lives at the billing edge

Outside the quota walls, 14,454 runs and zero silent failures inside any engine. At two quota walls, fifteen.

Every single loss we measured this quarter happened when a free tier ran out.

Two walls, opposite manners

Pipedream, July 2, 14:24 UTC. Credits ran out mid-measurement. From one event to the next, five seconds apart, delivery went from 100% to 0%. Its webhooks kept returning a success response for every event it then discarded: 14 of 14 lost, 95% CI 78 to 100%. No error, no queue, no replay, no notification. We re-tested 19 hours later: still accepting, still dropping.

Pabbly Connect, September 22, 04:41 UTC. Task allotment exhausted at 100. Within the same minute it sent two emails, named the workflow that had been refused, said the action step was not executed because the limit was exhausted, and held the trigger data in a pending state rather than dropping it.

For an account owner those are night and day, and Pabbly is obviously the better citizen. But notice what the workflow saw, which is what this metric measures: in both cases the webhook was accepted, no action ran, and no receipt arrived.

Pabbly's wall is loud to the account owner and silent to the automation. Pipedream's was silent to both. If your alerting watches the workflow rather than the billing mailbox, those two platforms produced the same event.

A third wall, found in our own logs

Make's free plan allows 1,000 credits a month, and one empty 30-minute poll of our scheduled workflow costs 2 of them. So that one workflow spends the month's allotment in about eleven days, and the scenario stops until the monthly reset. In Q3 it was dark from August 14 to September 2 and again from September 14 to the end of the quarter: 37 of the 90 days it was scheduled to run. Nothing errored and no email arrived. The only evidence was an absence of rows in our own poll log.

That is a third behaviour at a quota wall, unlike the other two: loud to nobody. It also turns a pricing-page caveat into a measured fact. Make's free tier cannot keep one always-on scheduled workflow running for a month. We measured one platform, so we will not say this of free tiers in general; on any plan that charges per operation, the arithmetic is worth doing before you trust a scheduled workflow to it, because the allotment can run out partway through the month without telling you.

What we would act on

If you run anything you care about on a metered tier, the failure semantics you are signing up for are: at the quota boundary, "accepted" stops meaning "delivered", and "scheduled" stops meaning "run". Not on some platforms. On every one we measured at its wall, in different ways that look the same from inside your workflow.

The mitigation is not a better platform. It is checking the destination rather than the platform's own status, on a schedule, so that "nothing arrived" is a number you can see.

One claim that went the vendor's way

Pabbly advertises that internal steps do not consume quota: triggers, filters, routers, formatters. That is normally a pricing-page assertion nobody outside the vendor can check. We can check it against a meter: 104 free internal tasks were consumed alongside the 100 billable ones, and not one of them was charged. The claim holds.

Latency

Median delivery, and the tail, in milliseconds:

Platformp50p95p99Worst observed
Pabbly8121,5713,7163,792
Make1,0091,3103,0374,044
n8n (self-hosted)1,0167,47720,07960,076
Pipedream2,7184,8955,2855,482
Zapier4,2959,81529,21167,570

Medians are comparable across platforms. Tails are not. The network path is identical for every platform; the computer is not. The hosted platforms run on their vendors' hardware. n8n runs on our own 1 GB free-tier machine.

Zapier is about four times slower than Make at the median, and its tail is far worse: p99 at 29 seconds, worst case 68. If you are chaining an automation into anything a human is waiting on, that tail is the number to design against, not the median.

Make has the tightest distribution of any platform we measured. p99 at 3 seconds against a 1 second median is remarkably flat.

n8n's tail is mostly ours. It matches Make at the median and then spreads badly. In October we watched why: between batches our machine pages n8n's memory out to a slow disk, and the next execution spends most of its time waiting to read it back. That is a fair picture of self-hosting on the smallest box you can get, and not a fair picture of n8n. We then tried three fixes, each with the pass mark written down before any data came in, and all three failed; the details are on /reliability. We have stopped tuning and leave the tail visible as the cost of a 1 GB machine.

Scheduler punctuality

For polling rather than webhook triggers, on a 30-minute interval:

Items deliveredPolls run vs scheduled
n8n383 of 383all of them
Make122 of 12357.7%, dark 37 of the 90 days it was scheduled to run, at the quota wall above

On the polls that did run, neither platform missed a tick. Make's one missed item changed during a nine-hour window when the scenario sat deactivated after we edited it, which Make does until you switch the scenario back on. Our misconfiguration, not a Make miss.

What went wrong on our side, and why it is in this report

A benchmark that only publishes other people's faults is marketing.

We broke our own receiver for 23 hours. On September 17 a config file added for type checking conflicted with how our endpoint is built, and every incoming receipt hit a function that would not start. All five platforms accepted and delivered normally the entire time; 101 of 121 runs found nothing at our end. We found out because a vendor emailed to say a step on their side was failing, which is a poor way to learn that a measurement system has stopped measuring. Those runs are excluded with the window and cause published, not deleted, because counting our own outage as the platforms' silent failures would be exactly the reclassification this metric exists to forbid.

We walked into Pabbly's quota wall with a guard written to prevent it. The guard's per-task cost model was right to within 3 of the cap. Its calendar was fiction: it assumed a monthly reset on the day the account was created, because that looked like the billing anchor, and nothing ever verified it. The allotment never reset. On the morning the quota was exhausted, the guard reported 10 tasks used against a real meter of 100.

We left Pipedream switched off for the whole quarter for no reason. When its credits ran out in July we paused it, with a note to resume once someone confirmed how often the allotment renews. Nobody did. It renews monthly, so August and September were both measurable and we measured neither. A probe written in September to detect exactly this kind of renewal was being dropped by two other checks before it could fire, while the log announced it every run. We found out on October 7, with one manual probe that took 45 seconds.

We called Make's poller "always-on" while it was off for 37 days. See the third quota wall above. The page said it; the log said otherwise; nobody had compared them.

Our own pages overstated our cleanliness. One said we had seen "exactly one organic rejection". True in July; by the end of the quarter there had been five. Others listed four platforms after Pabbly had been measured for weeks, or called measurement "continuous" on platforms where it had stopped. Each number or claim had been typed into a sentence rather than computed from the ledger. In early October we found six such statements, and five of the six made us look better than we were. They are now computed at build time or checked on every publish.

All five rejections were the self-hosted n8n box refusing a connection under load. Two have a recorded cause, ECONNRESET. Three do not, because we did not store rejection reasons until September 19. Three permanently undiagnosable events, in a project about undiagnosable events.

Since the quarter closed

Things a reader of this report should know that happened after September 30:

What we cannot tell you

Method, in one paragraph

Four fixed workflows per platform, fired on a schedule by GitHub Actions. Each workflow ends by posting a receipt to an endpoint we control. Every fired event records, before it is sent, how many receipts it should produce. Reconciliation compares expected against received. A run that produced fewer receipts than expected is a silent failure; a run refused at send time is a rejection and is excluded from both numerator and denominator; a run correctly stopped by a filter is excluded from the denominator; a run fired during an outage on our side is excluded and the window is published. Intervals are Wilson score intervals, because the normal approximation collapses to zero width at zero events and would let us print "0% plus or minus 0%", which is the exact claim this project exists to argue against.

The specification is published and versioned at /spec. The full per-run ledger is at /license under CC BY 4.0. Harness incidents, including ours, are listed on /reliability.

If you want the underlying data it is free and always will be. If you want something we have not published, or the right to use it without attribution, see /data-licensing.