osmose.validation.engine_equivalence

Ensemble equivalence of two engines that run the same model (C# engine spec, section 10.2).

Two in-process engines differ only in their random streams, so equivalence is judged against a CONTROL comparison: the Python engine against itself on disjoint seeds, through the same cells and statistics. A cell is one (metric, species): the per-replicate end-of-run values of two arms, compared on the log10 scale by a two-one-sided t-test (TOST) at a per-metric margin.

Acceptance (spec 10.2), all of:

  1. no DIVERGENT cell: |d| exceeds the margin AND the Bonferroni half-width t(1 - 0.10 / 2m) * se over the m counted cells. As specified this was the plain 90% interval; on EEC that rejected two IDENTICAL Python arms 78.5% of the time (a noisy cell excludes zero from its own 90% interval 10% of the time by construction, and no constant loosens it). Amended by the owner, 2026-11-18: Bonferroni measured 3.2% there (docs/perf/2026-11-18-csharp-ensemble-equivalence.md);

  2. equivalent cells of the treatment >= those of the control minus eq_slack;

  3. the treatment’s largest |d| <= max_ratio x the control’s, or every treatment cell within its margin;

  4. each cell’s variance ratio within var_factor (either way), except where the control cell fails it too;

  5. a cell that is one same number in every replicate of all three arms is vacuous: listed, not counted. Nothing draws for such a cell (a background species’ forced biomass, the zero yield of an unfished species, a value under the log floor), so it carries no information, and counting it made the number of equivalent cells read as more than it was. “The same number” is on the log scale and to rounding (SPREAD_FLOOR). Amended on the owner’s instruction, 2026-11-19; as specified only an all-zero yield was left out. A cell that is one number on both Python arms and NOT on the other engine is counted like any cell: that is a difference.

calibrate measures the rule’s false-fail rate for IDENTICAL engines by re-splitting the pooled Python replicates, and picks the first constants on a fixed ladder whose rate is at most 5%. It is run, and its constants frozen, before any treatment data are looked at.

Module Attributes

COLLAPSE

a collapsed species reads 1, not log(0).

FLOORS

Per-metric log floors.

SPREAD_FLOOR

A standard error below this (log10 units, ~2e-12 relative) is floating-point residue, not spread.

DIVERGENCE_ALPHA

each cell's divergence test runs at DIVERGENCE_ALPHA / m two-sided, m = the counted cells.

MARGINS

1.25x for stocks and flows, 1.05x for mean weight.

LADDER

Loosening order when the default constants false-fail identical engines too often.

Functions

acceptance(control, treatment[, margins, ...])

Spec 10.2's rule.

calibrate(arm_a, arm_b[, n_splits, margins, ...])

The rule's false-fail rate for identical engines, for every rung of ladder.

compare(x, y[, margins, with_ks])

Every (metric, species) cell both arms carry, as x against y.

tost(x, y, delta[, floor, with_ks])

Formal TOST of mean(log10 x) - mean(log10 y) against +-delta: returns (d, ci90_halfwidth, p_tost, equivalent, ks_p, var_ratio).

Classes

Cell(metric, species, d, ci, p, equivalent, ...)

One (metric, species) comparison.

Constants([eq_slack, max_ratio, var_factor])

The rule's tunable constants (spec 10.2: "minus one", 1.5, a factor of two).

Verdict(passed, failures, failed, divergent, ...)