# Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

arXiv: https://arxiv.org/abs/2506.19882 · HTML: https://mlsanity.com/paper/2506.19882/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 116 references that are in the map.
- First posted on arXiv 24 Jun 2025.

## Builds on (most cited first, 30 of 63)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?](https://mlsanity.com/paper/2003.11539/) (2020)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [How Does Batch Normalization Help Optimization?](https://mlsanity.com/paper/1805.11604/) (2018)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training](https://mlsanity.com/paper/2305.14342/) (2023)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [A Careful Examination of Large Language Model Performance on Grade School Arithmetic](https://mlsanity.com/paper/2405.00332/) (2024)
- [Lessons from the Trenches on Reproducible Evaluation of Language Models](https://mlsanity.com/paper/2405.14782/) (2024)
- [Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data](https://mlsanity.com/paper/2404.01413/) (2024)

## Built on by (most cited first, 7 of 7)

- [Insights from the ICLR Peer Review and Rebuttal Process](https://mlsanity.com/paper/2511.15462/) (2025)
- [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026)
- [DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)](https://mlsanity.com/paper/2507.07855/) (2025)
- [NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims](https://mlsanity.com/paper/2605.08192/) (2026)
- [An Investigation of the NeurIPS and ICML 2025 Position Tracks](https://mlsanity.com/paper/2608.16894/) (2026)
- [Training Witnesses: Trusting the Training without Trusting the Trainer](https://mlsanity.com/paper/2609.33915/) (2026)
- [How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models](https://mlsanity.com/paper/2609.33936/) (2026)

## Abstract (arXiv)

Science progresses by iteratively advancing and correcting humanity's understanding of the world. In machine learning (ML) research, rapid advancements have led to an explosion of publications, but have also led to misleading, incorrect, flawed or perhaps even fraudulent studies being accepted and sometimes highlighted at ML conferences due to the fallibility of peer review. While such mistakes are understandable, ML conferences do not offer robust processes to help the field systematically correct when such errors are made. This position paper argues that ML conferences should establish a dedicated "Refutations and Critiques" (R&C) Track. This R&C Track would provide a high-profile, reputable platform to support vital research that critically challenges prior research, thereby fostering a dynamic self-correcting research ecosystem. We discuss key considerations including track design, review principles, potential pitfalls, and provide an illustrative example submission concerning a recent ICLR 2025 Oral. We conclude that ML conferences should create official, reputable mechanisms to help ML research self-correct.