# An Investigation of the NeurIPS and ICML 2025 Position Tracks

arXiv: https://arxiv.org/abs/2608.16894 · HTML: https://mlsanity.com/paper/2608.16894/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 109 of its 193 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 30 of 109)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [LLM Generated Persona is a Promise with a Catch](https://mlsanity.com/paper/2503.16527/) (2025)
- [Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge](https://mlsanity.com/paper/2502.00561/) (2025)
- [Deep Learning is Not So Mysterious or Different](https://mlsanity.com/paper/2503.02113/) (2025)
- [Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks](https://mlsanity.com/paper/2502.14546/) (2025)
- [Stop Overvaluing Multi-Agent Debate -- We Must Rethink Evaluation and Embrace Model Heterogeneity](https://mlsanity.com/paper/2502.08788/) (2025)
- [Neither Valid nor Reliable? Investigating the Use of LLMs as Judges](https://mlsanity.com/paper/2508.18076/) (2025)
- [Medical Large Language Model Benchmarks Should Prioritize Construct Validity](https://mlsanity.com/paper/2503.10694/) (2025)
- [Position: AI Evaluation Should Learn from How We Test Humans](https://mlsanity.com/paper/2306.10512/) (2023)
- [Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards](https://mlsanity.com/paper/2505.04966/) (2025)
- [Position: Model Collapse Does Not Mean What You Think](https://mlsanity.com/paper/2503.03150/) (2025)
- [Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents](https://mlsanity.com/paper/2505.22655/) (2025)
- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [LLM Agents Should Employ Security Principles](https://mlsanity.com/paper/2505.24019/) (2025)
- [Large Language Models Miss the Multi-Agent Mark](https://mlsanity.com/paper/2505.21298/) (2025)
- [We Can't Understand AI Using our Existing Vocabulary](https://mlsanity.com/paper/2502.07586/) (2025)
- [We Should Chart an Atlas of All the World's Models](https://mlsanity.com/paper/2503.10633/) (2025)
- [Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints](https://mlsanity.com/paper/2503.01747/) (2025)
- [Algebra Unveils Deep Learning -- An Invitation to Neuroalgebraic Geometry](https://mlsanity.com/paper/2501.18915/) (2025)
- [Position: The Future of Bayesian Prediction Is Prior-Fitted](https://mlsanity.com/paper/2505.23947/) (2025)
- [Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects](https://mlsanity.com/paper/2505.18893/) (2025)
- [Position: Use Sparse Autoencoders to Discover Unknowns](https://mlsanity.com/paper/2506.23845/) (2025)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [Language model developers should report train-test overlap](https://mlsanity.com/paper/2410.08385/) (2024)
- [Position: The Most Expensive Part of an LLM should be its Training Data](https://mlsanity.com/paper/2504.12427/) (2025)

## Abstract (arXiv)

ML venues shape what kinds of research claims become legible to reviewers and what forms of evidence count as rigorous. The NeurIPS and ICML Position Paper Tracks were created for agenda-setting work, making their early composition worth auditing. This paper argues that the publicly accessible 2025 reviewed pool is dominated by reformist critique, and that the track should explicitly solicit direction-setting work alongside, not in place of, the reformist critiques it already hosts well. We audit every accessible submission to the NeurIPS 2025 and ICML 2025 Position Tracks under a pre-specified rubric, and compare the resulting pattern with a reference class of widely recognized agenda-shifting ML papers. Three-quarters of audited submissions critique an existing benchmark, evaluation, or methodology; these papers score highly on our artifact-coupling rubric, but evidentiary depth does not predict reviewer rating. The reference class (AlexNet, the Transformer, Concrete Problems in AI Safety, and others) differs from the accessible reviewed pool in artifact kind: agenda-shifting papers typically gave the field something new to build on, test against, or contest, such as a measurement protocol, benchmark proposal, toy implementation, dataset card, audit template, or falsifiable experimental program. We close with four CFP-level interventions aimed at broadening the submission mix without displacing the critiques the track already hosts well.