mlsanityData to 7 Oct 2026

How Open Must Language Models be to Enable Reliable Scientific Inference?

arXiv 2603.26539 ↗First posted 27 Mar 2026cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

How does the extent to which a model is open or closed impact the scientific inferences that can be drawn from research that involves it? In this paper, we analyze how restrictions on information about model construction and deployment threaten reliable inference. We argue that current closed models are generally ill-suited for scientific purposes, with some notable exceptions, and discuss ways in which the issues they present to reliable inference can be resolved or mitigated. We recommend that when models are used in research, potential threats to inference should be systematically identified along with the steps taken to mitigate them, and that specific justifications for model selection should be provided.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 105 of its 153 references that are in the map. First posted on arXiv 27 Mar 2026.

Builds on

105 of 153 references.

References · 105

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
38 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2018Model Cards for Model Reportingcited by 1,068
  6. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  8. 2022In-context Learning and Induction Headscited by 930
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  12. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  13. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  14. 2023Cognitive Architectures for Language Agentscited by 422
  15. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  16. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  17. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  18. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  19. 20242 OLMo 2 Furiouscited by 302
  20. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  21. 2023How is ChatGPT's behavior changing over time?cited by 265
  22. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  23. 2017OpenNMT: Open-source Toolkit for Neural Machine Translationcited by 251
  24. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  25. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  26. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  27. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  28. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  29. 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
  30. 2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
  31. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
  32. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  33. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 119
  34. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  35. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  36. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  37. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  38. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 2
  2. 2026Epistemic orientation predicts legislative effectiveness among members of the US Congresscited by 0