mlsanityData to 14 Oct 2026

Predictable Compression Failures: Order Sensitivity and Information Budgeting for Evidence-Grounded Binary Adjudication

arXiv 2509.11208 ↗First posted 14 Sep 2025stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Transformers used for evidence-grounded binary adjudication (e.g., support/refute, yes/no, or verifier-backed pass/fail decisions) can be sensitive to the order in which exchangeable evidence is presented, producing dispersion across permutations and unreliable attempted answers under a verifier-relative Bernoulli predicate. We treat evidence order as a nuisance variable and formalize an expectation-realization gap: next-token training can minimize expected conditional description length over orderings while a fixed ordering remains position-sensitive. Our Quantified Martingale Violation (QMV) bound predicts the dispersion induced by adjacent-rank positional sensitivity, with O(log n) growth in the harmonic regime; our Expectation-level Decompression Law (EDFL) specializes a KL convexity/data-processing bound to Bernoulli predicates, yielding Bits-to-Trust (B2T), Risk-of-Hallucination (RoH), and an Information Sufficiency Ratio (ISR) gate for answer/abstain decisions. On 3,059 grounded items from FEVER, HotpotQA, NQ-Open, PopQA, and Controls, we observe logarithmic dispersion and positive Jensen gains from uniform permutation mixtures. In one pre-specified held-out audit (528 items), the analytically fixed ISR=1 gate attains 0.0-0.7% hallucination with 20.6-27.9% abstention (95% CIs), supporting the operating point without claiming universal calibration across all model families or unrestricted generation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 14 of its 23 references that are in the map. First posted on arXiv 14 Sep 2025.

Builds on

14 of 23 references.

References · 14

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2017Selective Classification for Deep Neural Networkscited by 852
  6. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  7. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  8. 2023Language Modeling Is Compressioncited by 225
  9. 2023Calibrated Language Models Must Hallucinatecited by 115
  10. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  11. 2024Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspectivecited by 57
  12. 2024Base of RoPE Bounds Context Lengthcited by 52
2 more
  1. 2025Can Transformers Learn Full Bayesian Inference in Context?cited by 31
  2. 2024Beyond Self-Consistency: Ensemble Reasoning Boosts Consistency and Accuracy of LLMs in Cancer Stagingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testingcited by 2
  2. 2026Truth as a Compression Artifact in Language Model Trainingcited by 1