mlsanityData to 14 Oct 2026

Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

arXiv 2506.19882 ↗First posted 24 Jun 2025cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Science progresses by iteratively advancing and correcting humanity's understanding of the world. In machine learning (ML) research, rapid advancements have led to an explosion of publications, but have also led to misleading, incorrect, flawed or perhaps even fraudulent studies being accepted and sometimes highlighted at ML conferences due to the fallibility of peer review. While such mistakes are understandable, ML conferences do not offer robust processes to help the field systematically correct when such errors are made. This position paper argues that ML conferences should establish a dedicated "Refutations and Critiques" (R&C) Track. This R&C Track would provide a high-profile, reputable platform to support vital research that critically challenges prior research, thereby fostering a dynamic self-correcting research ecosystem. We discuss key considerations including track design, review principles, potential pitfalls, and provide an illustrative example submission concerning a recent ICLR 2025 Oral. We conclude that ML conferences should create official, reputable mechanisms to help ML research self-correct.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 63 of its 116 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

63 of 116 references.

References · 63

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Mistral 7Bcited by 3,164
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  10. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  11. 2018Hierarchical Neural Story Generationcited by 1,193
  12. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
38 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  5. 2019A Closer Look at Few-shot Classificationcited by 684
  6. 2014Distributed Representations of Sentences and Documentscited by 678
  7. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  9. 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
  10. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  11. 2018How Does Batch Normalization Help Optimization?cited by 357
  12. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  13. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  14. 2023Self-Consuming Generative Models Go MADcited by 250
  15. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  16. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  17. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  18. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  19. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  20. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  21. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  22. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 120
  23. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  24. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 88
  25. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  26. 2024Best-of-N Jailbreakingcited by 76
  27. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 57
  28. 2025Do Large Language Model Benchmarks Test Reliability?cited by 54
  29. 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 53
  30. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  31. 2017MagNet and "Efficient Defenses Against Adversarial Attacks" are Not Robust to Adversarial Examplescited by 46
  32. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  33. 2024Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?cited by 40
  34. 2020Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directionscited by 34
  35. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 34
  36. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  37. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 26
  38. 2022No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"cited by 23

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.