SFR v1.0: the Silent Failure Rate specification
How to measure it, and what a report has to contain before its number means anything · v1.0, released 2026-09-09 · CC BY 4.0
Read the specification on GitHub · machine-readable form · cite as SFR v1.0, https://github.com/benchtruth/sfr-spec
A silent failure is a unit of work a platform accepted, told you nothing was wrong about, and did not complete. This specification says how to measure how often that happens, and, more importantly, what a published rate has to carry before anyone should believe it.
It exists because the number is easy to get wrong in ways that flatter whoever is reporting it. Silent failure rates are small, so the usual statistical shortcuts break. Failures can be moved in or out of the count by reclassifying them. And a rate published without its denominator cannot be checked, compared, or distinguished from a rate that was simply asserted.
This is not a tool. Nothing here is installed or run. It constrains how a measurement is made and reported, and does not require any particular software, including ours.
The metric
missed + partial
SFR = ------------------------------------------
runs accepted and expected to produce output- missed: accepted, nothing arrived at the destination.
- partial: accepted, some but not all of the expected effects arrived.
- Runs correctly stopped by a filter are not failures and are not in the denominator. They are reported separately.
The send-outcome exclusion
Events a platform refused at the moment of delivery are excluded from both the numerator and the denominator, and reported as a separate count. A non-2xx response, a timeout, or a dropped connection at the point of sending is a loud failure: the sender sees it immediately, can retry it, and no run was created.
That is the opposite property from the one this metric measures. Combining the two produces a number that cannot distinguish "the platform told you and you can retry" from "the platform did not tell you and the data is gone". This is the load-bearing rule of the specification, and it is the one that stops a rate being tuned by reclassification. Longer treatment on silent failure vs rejected webhook.
A worked example from our own measurements: in a five-hour sustained-load run of 2,880 events, 4 were refused at send and 0 accepted events failed to arrive. The conformant report is 0 silent failures in 2,876 runs, with 4 rejections at send reported alongside. Writing "4 failures in 2,880" would be wrong in both directions: it inflates the failure rate with events the sender saw immediately, and it hides the fact that nothing accepted was lost. The full run is on the soak page.
Conformance levels
A level is a property of a published measurement, not of an organisation. "We are SFR-conformant" means nothing. SFR-conformant (L1), SFR v1.0, attached to a specific report, is a claim a reader can check.
Conformance is self-declared. There is no certifying body, no registry and no audit. We do not review claims and will not arbitrate disputes about them. The levels are useful only because every requirement is checkable by a reader of the report: if a claim is false, the report will not contain what the level requires.
| Level | Requires | Reachable by |
|---|---|---|
| L1 Reported | Per-event identity · destination-side verification · the send-outcome exclusion · denominator stated · Wilson 95% interval with no bare zero · as-of date and window · workload described | Anyone already running automations. No harness, no paid plans, no experiment. Its bar is reporting discipline, not effort. |
| L2 Measured | L1 · independent trigger · both endpoints under your control · reference workloads or a documented subset · method published · per-run data available | Anyone willing to run a controller and a destination for a few weeks. |
| L3 Stress | L2 · destination-outage probe · success-wrapped-failure probe · sustained-load probe, each published including when the result is that nothing interesting happened | Hard. Requires deliberately breaking your own destination and publishing what happened. |
If a requirement is not met, do not claim the level. Claim the level below and state what is missing. A partial claim is worth more than an inflated one, and it is the only kind that survives a reader checking it.
What a report has to carry
The requirement that does the most work is the least glamorous: every published rate carries its denominator, its interval, its as-of date and its workload in the same sentence. Not in a footnote, not on a linked methodology page.
That reads as pedantry until you follow a number through extraction: from a table into a summary, from a summary into a quotation, from a quotation into someone's slide. Every step drops whatever was not adjacent to the number. The only qualifier that survives is the one inside the sentence.
Three more that are routinely omitted:
- Say whether the figure is alive. Either it is refreshed, and how often, or it is a one-off snapshot that will not be updated. A one-off measurement is legitimate; a one-off presented in the present tense two years later is not, and the difference is one sentence.
- Quoting a rate makes you a publisher of it. Carry the original's qualifiers and its as-of date, not the date you quoted it. Do not aggregate rates across workloads, platforms or windows: there is no such thing as the industry's silent failure rate.
- Do not put a measured rate in a table beside rates that have no denominator without marking which is which. Once both are rendered as a percentage in a cell they look identical, and the reader supplies a comparability that was never there.
A worked example you can check
Our own reliability scoreboard is written to be L3-conformant, and it is here as a worked example rather than as a badge. As of 2026-09-09 it reports 12,717 monitored runs with per-platform denominators, Wilson bounds and an as-of date on every figure, for example Zapier at 0 silent failures in 840 runs, an upper bound of 0.46% at 95% confidence. Rejections at send are excluded and counted separately. The three L3 probes are published as outage semantics, the 200 OK error and sustained load.
Check it against the requirements above. If something we publish does not meet a level we claim, that is a defect in our reporting and we would like to know: @benchtruth. Corrections in this project are additive and published where the original figure appeared, which is the same rule the specification asks of everyone else.
Status, and what is deliberately not here
- Released, and stable. The specification is at github.com/benchtruth/sfr-spec, tagged
v1.0. Published text is not edited in place, so a claim of conformance to v1.0 stays valid. Machine-readable form:sfr-v1.json. - Reference implementation: planned, not yet published. The specification constrains how a measurement is made and reported. It does not require any particular software, including ours, and we are not going to pretend the code is out when it is not.
- One requirement is reserved. Section 6.6, on reporting a result that contradicts the prevailing claim, is left empty on purpose. The rule that belongs there is being drawn from observed evidence rather than invented, and that evidence is not complete. It will be added as an erratum.
- The specification is additive. Published text is not edited in place; corrections and clarifications are appended with their own dates, so a claim of conformance to a version stays valid.
- No support. There is no certifying body, no registry, and no undertaking to extend the reference workloads to other shapes. This is maintained by one person alongside other work, and the guarantees are small so that they can be kept.
Questions
What is the Silent Failure Rate?
The share of runs an automation platform accepted, did not complete, and did not report as failed. Formally: (missed + partial) divided by the runs the platform accepted and that the workload expected to produce output. Three conditions must all hold for a run to count: the platform accepted it and answered successfully, the expected effect never appeared at the destination, and nothing in the platform's normal reporting told the operator. The third condition is what makes it silent and is why it needs its own metric: a failure that announces itself is an operational event you retry, while one that does not is a data-integrity problem you discover weeks later.
Why are rejected webhooks excluded from the Silent Failure Rate?
Because the sender already knows. When a platform refuses an event at the moment of delivery, with a non-2xx status, a timeout or a dropped connection, the caller sees an error immediately, can log it, alert on it and retry it, and no run was ever created. That is the opposite property from the one the metric exists to measure. Combining them produces a number that cannot distinguish a loud recoverable failure from a hidden unrecoverable one. SFR v1.0 therefore excludes them from both the numerator and the denominator and requires them to be reported as a separate count. This is the load-bearing rule of the metric: it is what stops a measurement being tuned by reclassification.
Who can claim SFR conformance, and who certifies it?
Anyone, and nobody. Conformance is self-declared. There is no certifying body, no registry and no audit, and BenchTruth does not review claims or arbitrate disputes about them. The levels work because every requirement is checkable by a reader of the report itself: if a claim is false, the report will not contain what the level requires, and anyone can see that. A conformance claim is a property of a published measurement rather than of a company, so 'we are SFR-conformant' means nothing, while 'the figures in this report are SFR-conformant (L1), SFR v1.0' is a statement a reader can check.
What does L1 conformance actually require?
No test harness, no purchased subscriptions and no controlled experiment. L1 is reachable by anyone already running automations, because its bar is reporting discipline rather than effort. It requires seven things: every event carries an identifier preserved to the destination; success is verified at the destination rather than read from the platform's run history; events refused at send are excluded from both numerator and denominator and counted separately; the denominator is stated; a Wilson 95% interval accompanies the rate and a bare 0% is never published; the as-of date and window are stated; and the workload is described. Most published silent-failure figures today fail L1 on the reporting requirements alone, because they arrive without a denominator.