# The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?

arXiv: https://arxiv.org/abs/2401.02452 · HTML: https://mlsanity.com/paper/2401.02452/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 74 references that are in the map.
- First posted on arXiv 4 Jan 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Deep Learning in Neural Networks: An Overview](https://mlsanity.com/paper/1404.7828/) (2014)
- [Release Strategies and the Social Impacts of Language Models](https://mlsanity.com/paper/1908.09203/) (2019)
- [Training Neural Networks with Fixed Sparse Masks](https://mlsanity.com/paper/2111.09839/) (2021)
- [Predictability and Surprise in Large Generative Models](https://mlsanity.com/paper/2202.07785/) (2022)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing](https://mlsanity.com/paper/2007.06225/) (2020)
- [The Cost of Training NLP Models: A Concise Overview](https://mlsanity.com/paper/2004.08900/) (2020)
- [Luna: Linear Unified Nested Attention](https://mlsanity.com/paper/2106.01540/) (2021)
- [Pre-training via Paraphrasing](https://mlsanity.com/paper/2006.15020/) (2020)
- [The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations](https://mlsanity.com/paper/2106.00786/) (2021)
- [How to Train BERT with an Academic Budget](https://mlsanity.com/paper/2104.07705/) (2021)
- [The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research](https://mlsanity.com/paper/2010.15581/) (2020)
- [Structured access: an emerging paradigm for safe AI deployment](https://mlsanity.com/paper/2201.05159/) (2022)
- [Varuna: Scalable, Low-cost Training of Massive Deep Learning Models](https://mlsanity.com/paper/2111.04007/) (2021)
- [The Role of Cooperation in Responsible AI Development](https://mlsanity.com/paper/1907.04534/) (2019)
- [A narrowing of AI research?](https://mlsanity.com/paper/2009.10385/) (2020)
- [Artificial Intelligence for Social Good](https://mlsanity.com/paper/1901.05406/) (2019)
- [Defining the role of open source software in research reproducibility](https://mlsanity.com/paper/2204.12564/) (2022)

## Built on by (most cited first, 20 of 20)

- [Efficient Benchmarking of AI Agents](https://mlsanity.com/paper/2603.23749/) (2026)
- [Dissecting Submission Limit in Desk-Rejections: A Mathematical Analysis of Fairness in AI Conference Policies](https://mlsanity.com/paper/2502.00690/) (2025)
- [EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference](https://mlsanity.com/paper/2502.04700/) (2025)
- [HRM-Text: Efficient Pretraining Beyond Scaling](https://mlsanity.com/paper/2605.20613/) (2026)
- [Content Generation Models in Computational Pathology: A Comprehensive Survey on Methods, Applications, and Challenges](https://mlsanity.com/paper/2505.10993/) (2025)
- [Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Papers](https://mlsanity.com/paper/2511.21739/) (2025)
- [Trends in AI Supercomputers](https://mlsanity.com/paper/2504.16026/) (2025)
- [Agentic Inequality](https://mlsanity.com/paper/2510.16853/) (2025)
- [Limits to AI Growth: The Ecological and Social Consequences of Scaling](https://mlsanity.com/paper/2501.17980/) (2025)
- [MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting](https://mlsanity.com/paper/2601.09085/) (2026)
- [Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism](https://mlsanity.com/paper/2602.04870/) (2026)
- [LLM Vocabulary Compression for Low-Compute Environments](https://mlsanity.com/paper/2411.06371/) (2024)
- [Accept More, Reject Less: Reducing up to 19% Unnecessary Desk-Rejections over 11 Years of ICLR Data](https://mlsanity.com/paper/2506.20141/) (2025)
- [Inference Time Debiasing Concepts in Diffusion Models](https://mlsanity.com/paper/2508.14933/) (2025)
- [Can David Beat Goliath? On Multi-Hop Reasoning with Resource-Constrained Agents](https://mlsanity.com/paper/2601.21699/) (2026)
- [A computationally frugal open-source foundation model for thoracic disease detection in lung cancer screening programs](https://mlsanity.com/paper/2507.01881/) (2025)
- [Structural shifts in institutional participation and collaboration within the AI arXiv preprint research ecosystem](https://mlsanity.com/paper/2602.03969/) (2026)
- [Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation](https://mlsanity.com/paper/2602.18483/) (2026)
- [More Computational Resources Do Not Ensure Higher Scholarly Impact: Evidence from Leading NLP Conference Papers](https://mlsanity.com/paper/2608.21806/) (2026)
- [AI in Science: Early Insights](https://mlsanity.com/paper/2609.28504/) (2026)

## Abstract (arXiv)

There are pronounced differences in the extent to which industrial and academic AI labs use computing resources. We provide a data-driven survey of the role of the compute divide in shaping machine learning research. We show that a compute divide has coincided with a reduced representation of academic-only research teams in compute intensive research topics, especially foundation models. We argue that, academia will likely play a smaller role in advancing the associated techniques, providing critical evaluation and scrutiny, and in the diffusion of such models. Concurrent with this change in research focus, there is a noticeable shift in academic research towards embracing open source, pre-trained models developed within the industry. To address the challenges arising from this trend, especially reduced scrutiny of influential models, we recommend approaches aimed at thoughtfully expanding academic insights. Nationally-sponsored computing infrastructure coupled with open science initiatives could judiciously boost academic compute access, prioritizing research on interpretability, safety and security. Structured access programs and third-party auditing may also allow measured external evaluation of industry systems.