Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track
Not on the map. Why?
Abstract (arXiv)
Science progresses by iteratively advancing and correcting humanity's understanding of the world. In machine learning (ML) research, rapid advancements have led to an explosion of publications, but have also led to misleading, incorrect, flawed or perhaps even fraudulent studies being accepted and sometimes highlighted at ML conferences due to the fallibility of peer review. While such mistakes are understandable, ML conferences do not offer robust processes to help the field systematically correct when such errors are made. This position paper argues that ML conferences should establish a dedicated "Refutations and Critiques" (R&C) Track. This R&C Track would provide a high-profile, reputable platform to support vital research that critically challenges prior research, thereby fostering a dynamic self-correcting research ecosystem. We discuss key considerations including track design, review principles, potential pitfalls, and provide an illustrative example submission concerning a recent ICLR 2025 Oral. We conclude that ML conferences should create official, reputable mechanisms to help ML research self-correct.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 63 of its 116 references that are in the map. First posted on arXiv 24 Jun 2025.
Builds on
63 of 116 references.
References · 63
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Mistral 7Bcited by 3,164
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2016Deep Learning with Differential Privacycited by 2,728
- 2020Learning to summarize from human feedbackcited by 2,588
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
- 2018Hierarchical Neural Story Generationcited by 1,193
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
38 more
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
- 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
- 2019A Closer Look at Few-shot Classificationcited by 684
- 2014Distributed Representations of Sentences and Documentscited by 678
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
- 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
- 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
- 2018How Does Batch Normalization Help Optimization?cited by 357
- 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
- 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
- 2023Self-Consuming Generative Models Go MADcited by 250
- 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
- 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
- 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
- 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
- 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
- 2024Theoretical guarantees on the best-of-n alignment policycited by 124
- 2022Truncation Sampling as Language Model Desmoothingcited by 123
- 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 120
- 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
- 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
- 2024Model Collapse Demystified: The Case of Regressioncited by 82
- 2024Best-of-N Jailbreakingcited by 76
- 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
- 2025Do Large Language Model Benchmarks Test Reliability?cited by 54
- 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
- 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
- 2017MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examplescited by 46
- 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
- 2024Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?cited by 40
- 2020Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directionscited by 34
- 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 34
- 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
- 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
- 2022No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"cited by 23
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2025: 2; 2026: 5.
7 papers.
- 2025Insights from the ICLR Peer Review and Rebuttal Processcited by 11
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
- 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
- 2026NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claimscited by 0
- 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
- 2026Training Witnesses: Trusting the Training without Trusting the Trainercited by 0
- 2026How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Modelscited by 0
- 2026How Strong Is the Evidence for the Artificial Hivemind? Reevaluating Evidence for the Open-Ended Homogeneity of Language Models27 Sep 2026
- 2026Training Witnesses: Trusting the Training without Trusting the Trainer27 Sep 2026
- 2026An Investigation of the NeurIPS and ICML 2025 Position Tracks6 Jul 2026
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap29 Jun 2026
- 2026NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims5 May 2026
- 2025Insights from the ICLR Peer Review and Rebuttal Process19 Nov 2025
- 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)10 Jul 2025