Run a batch pipeline several times on one input and report, per step, how often it failed to give the same answer. Running it five times is the easy half. Deciding whether two outputs are the same answer7 (source 7: twicerun, repository README on GitHub)twicerun, README is the project: DuckDB adds the terms of a parallel sum in whatever order the threads finish in, float addition is not associative, and so a plain two-run diff reports hundreds of findings on correct code.
On the reference pipeline it separates the five steps that diverged from the three that did not, then re-runs each divergent step at one thread to say whether the thread count was the explanation or whether something else is going on. It stops there rather than guessing between a clock read, a data-dependent branch and appended state.
The part I care about is what it refuses to say. A step that agreed with itself every time is reported as stable on this input, not as correct, and one such step is a planted bug that only moves once the input is stressed8 (source 8: twicerun, the annotated run on the reference pipeline, in the README)twicerun, the reference run in full. Where four clean comparisons are all the evidence there is, it prints the bound they support, a per-comparison divergence rate no higher than 53 percent, rather than a green tick.
