mlsanityData to 7 Oct 2026

Epistemic Compression: The Case for Deliberate Ignorance in High-Stakes AI

arXiv 2603.25033 ↗First posted 26 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Foundation models excel in stable environments, yet often fail where reliability matters most: medicine, finance, and policy. This Fidelity Paradox is not just a data problem; it is structural. In domains where rules change over time, extra model capacity amplifies noise rather than capturing signal. We introduce Epistemic Compression: the principle that robustness emerges from matching model complexity to the shelf life of the data, not from scaling parameters. Unlike classical regularization, which penalizes weights post hoc, Epistemic Compression enforces parsimony through architecture: the model structure itself is designed to reduce overfitting by making it architecturally costly to represent variance that exceeds the evidence in the data. We operationalize this with a Regime Index that separates Shifting Regime (unstable, data-poor; simplicity wins) from Stable Regime (invariant, data-rich; complexity viable). In an exploratory synthesis of 15 high-stakes domains, this index was concordant with the empirically superior modeling strategy in 86.7% of cases (13/15). High-stakes AI demands a shift from scaling for its own sake to principled parsimony.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 86 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

38 of 86 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  8. 2019Invariant Risk Minimizationcited by 1,832
  9. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  10. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  11. 2020In Search of Lost Domain Generalizationcited by 1,042
  12. 2015Deep Learning and the Information Bottleneck Principlecited by 793
26 more
  1. 2021Revisiting Deep Learning Models for Tabular Datacited by 725
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  3. 2024KAN: Kolmogorov-Arnold Networkscited by 634
  4. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  5. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  6. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  7. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
  8. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  9. 2020Liquid Time-constant Networkscited by 160
  10. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  11. 2020Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scanscited by 158
  12. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  13. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 141
  14. 2018Anchor regression: heterogeneous data meets causalitycited by 116
  15. 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
  16. 2020Stable Prediction with Model Misspecification and Agnostic Distribution Shiftcited by 62
  17. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  18. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  19. 2025Scaling Laws Are Unreliable for Downstream Tasks: A Reality Checkcited by 27
  20. 2023White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?cited by 22
  21. 2021Effective dimension of machine learning modelscited by 14
  22. 2025Cascaded Language Models for Cost-effective Human-AI Decision-Makingcited by 11
  23. 2025Beyond the Failures: Rethinking Foundation Models in Pathologycited by 6
  24. 2025Causality can systematically address the monsters under the bench(marks)cited by 4
  25. 2025A Conjecture on a Fundamental Trade-Off between Certainty and Scope in Symbolic and Generative AIcited by 4
  26. 2025The Information-Theoretic Imperative: Compression and the Epistemic Foundations of Intelligencecited by 1

Built on by

Not cited yet.