BENCHMARK VALIDITY NOTICE · 2026-07-18
The D-041 cohort is quarantined
All 20 historical results are retained as raw evidence but are ineligible for ranking or capability claims until they are rerun on Harness v2.
What the audit reproduced
Every one of the twelve published raw 1/68 outcomes occurred only at episode index 42, adversarial / moving_disturbance. With seed 84, the server's random walk moves the ungrasped object inside the 0.08 m goal radius at control step 156. The old success predicate checked only object-to-target distance, so environment motion could be credited to the policy.
What the historical runs mean
They record how the old integrations behaved. They do not establish policy success rates, production readiness, safety, or architecture-independent transfer. A raw 0/68 is not promoted to a valid zero; it remains unqualified until both the adapter and harness pass validation.
Additional qualification failures
- The positive control covered three nominal trials rather than every scored scenario.
- Published files used multiple schemas and lacked one immutable harness identity.
- Many episodes lacked IDs and seeds, and clients used unequal step budgets.
- Some adapters duplicated camera inputs or converted inference errors to zero actions.
- The runner could erase a non-zero client exit code and the publisher accepted unchecked JSON.
Publication reopening gates
- Verified grasp, lift, transport, release and stable-placement transitions.
- Zero-action negative controls and per-scenario positive controls.
- Model-specific adapter preflight with no hidden inference or protocol errors.
- Immutable run provenance and exactly 68 unique, seeded episode records.
- Strict semantic validation before any result can reach the publisher.