mlsanityData to 14 Oct 2026

Independent Verification Paths Are Not Independent: A Case Study of Common-Mode Failure in a Satellite Catalogue Pipeline

arXiv 2609.37603 ↗First posted 29 Sep 2026cs.SE cs.AI cs.DBcited by 0

Not on the map. Why?

Abstract (arXiv)

A common safeguard for a data pipeline is redundant computation: derive each published number by two routes built on different technology and refuse to exit when they disagree. We report one such gate failing, in a cross-catalogue integrity study of two open registers of Earth-orbiting objects. A gate comparing a set-based Python path with SPARQL queries over the emitted RDF graph printed ALL CROSS-CHECKS AGREE on seven counts. Three were wrong, one overstated more than fourfold (932 against 220). Both paths imported the same constants, which encoded a misreading of the source's status vocabulary, so the error was common-mode and the gate could not see it. We give the mechanism, an object-level ledger reconciling every figure, and three checks that go back to the source's documentation, measured on the defective code and on its correction. We then checked that correction against each object's phase history, held in a source file the pipeline never read. The correction was also wrong: 42 of its 261 disagreements are artefacts, and none of our three checks flagged them. Finally, in a controlled replication with three pinned models and tools disabled, 72 of 75 paths generated on request as independent checks computed the defective count, 29 of 30 even when the prompt carried the source's own definitions of the codes. The evidence is one pipeline and one defect family. Within it, redundancy verified implementation, and the errors that reached publication were errors of meaning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 4 of its 12 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

4 of 12 references.

References · 4

  1. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  2. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  3. 2022CodeT: Code Generation with Generated Testscited by 384
  4. 2024Do LLMs generate test oracles that capture the actual or the expected program behaviour?cited by 11

Built on by

Not cited yet.