Position: Ideas Should be the Center of Machine Learning Research
Not on the map. Why?
Abstract (arXiv)
Machine learning research increasingly bifurcates into two disconnected modes: benchmark-driven engineering that prioritizes metrics over understanding, and idealized theory that often fails to transfer to modern systems. In this position paper, we argue that the field focuses too heavily on these endpoints, neglecting the central scientific object: the idea. We propose an Ideas First framework in which ideas are valued for the behavioral signatures they predict in modern models, and these signatures are tested through tailored experiments designed to detect the relevant patterns rather than to win leaderboards. This shift not only bridges the gap between theory and practice but also promotes equity by removing the "complexity premium," enabling rigorous scientific contributions from researchers with modest computational, financial, and human resources. Ultimately, we advocate for a research culture centered on ideas, treating benchmarks and theorems as instruments for testing mechanistic hypotheses rather than as ends in themselves.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 14 May 2026.
Builds on
41 of 65 references.
References · 41
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
29 more
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
- 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
- 2017Spectrally-normalized margin bounds for neural networkscited by 686
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2019Green AIcited by 627
- 2019Benign Overfitting in Linear Regressioncited by 549
- 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
- 2015Norm-Based Capacity Control in Neural Networkscited by 299
- 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
- 2021Accounting for Variance in Machine Learning Benchmarkscited by 201
- 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
- 2021The Benchmark Lotterycited by 136
- 2018Troubling Trends in Machine Learning Scholarshipcited by 108
- 2020Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligencecited by 101
- 2021A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Levelcited by 96
- 2020The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Researchcited by 67
- 2025The Leaderboard Illusioncited by 66
- 2021Scientific Machine Learning Benchmarkscited by 60
- 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
- 2021A Survey on Green Deep Learningcited by 52
- 2022Sources of Irreproducibility in Machine Learning: A Reviewcited by 32
- 2021The Limits of Global Inclusion in AI Developmentcited by 14
- 2024From Protoscience to Epistemic Monoculture: How Benchmarking Set the Stage for the Deep Learning Revolutioncited by 12
- 2020We Should at Least Be Able to Design Molecules That Dock Wellcited by 9
- 2025Dynamics of Spontaneous Topic Changes in Next Token Prediction with Self-Attentioncited by 2
- 2025How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluationcited by 2
Built on by
Not cited yet.