When Scientific Contradictions Are Lost in Translation
Not on the map. Why?
Abstract (arXiv)
Two scientific findings can disagree without contradicting each other. Determining whether they conflict requires knowing whether they describe comparable measurements. We study how language models behave at this decision point. In a controlled task, we generate an unsatisfiable XOR constraint system and translate its constraints into scientific reports from different laboratories. One assignment satisfies more constraints, while another satisfies fewer but better matches expected biology. This creates a simple dilemma: does the model choose the assignment that best fits the constraints, or the one that better matches biological expectations? When the constraints are stated directly, GPT-5.6 Sol and Claude Opus 5 recover the best-supported assignment in 90% and 96% of cases, respectively. In scientific prose, however, the models behave differently. Claude Opus 5 often prefers the biologically expected assignment. Removing that biological preference increases recovery of the better-supported assignment from 27% to 79% (p<.001); recovery reaches 92% when the same Biology-favored record is accompanied by a formalization request and an explicit paired-design cue (p<.001). GPT-5.6 Sol is less sensitive, with neither corresponding change reaching statistical significance. These results suggest that reliable scientific verification depends not only on formal reasoning, but also on how models decide which findings should be compared and what relations they imply.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 11 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
9 of 11 references.
References · 9
- 2020Fact or Fiction: Verifying Scientific Claimscited by 546
- 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
- 2022Autoformalization with Large Language Modelscited by 252
- 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
- 2023SatLM: Satisfiability-Aided Language Models Using Declarative Promptingcited by 100
- 2023BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Informationcited by 42
- 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 25
- 2025SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verificationcited by 16
- 2025How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effectscited by 3
Built on by
Not cited yet.