mlsanityData to 7 Oct 2026

The Hardware Lottery

arXiv 2009.06489 ↗First posted 14 Sep 2020cs.CY cs.AI cs.ARcited by 160

Not on the map. Why?

Abstract (arXiv)

Hardware, systems and algorithms research communities have historically had different incentive structures and fluctuating motivation to engage with each other explicitly. This historical treatment is odd given that hardware and software have frequently determined which research ideas succeed (and fail). This essay introduces the term hardware lottery to describe when a research idea wins because it is suited to the available software and hardware and not because the idea is superior to alternative research directions. Examples from early computer science history illustrate how hardware lotteries can delay research progress by casting successful ideas as failures. These lessons are particularly salient given the advent of domain specialized hardware which make it increasingly costly to stray off of the beaten path of research ideas. This essay posits that the gains from progress in computing are likely to become even more uneven, with certain research directions moving into the fast-lane while progress on others is further obstructed.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 23; 2023: 32; 2024: 29; 2025: 25; 2026: 27.

160 papers.

Sort:

By area:

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2024The Platonic Representation Hypothesiscited by 362
  7. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  8. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  9. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  10. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
  11. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  12. 2021The Benchmark Lotterycited by 133
38 more
  1. 2021Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systemscited by 125
  2. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  3. 2021The Efficiency Misnomercited by 98
  4. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  5. 2023Everything is Connected: Graph Neural Networkscited by 85
  6. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  7. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  8. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  9. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  10. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
  11. 2023Intriguing Properties of Quantization at Scalecited by 48
  12. 2022Message passing all the way upcited by 47
  13. 2024Training of Physical Neural Networkscited by 47
  14. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  15. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  16. 2022evosax: JAX-based Evolution Strategiescited by 39
  17. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  18. 2024Artificial Kuramoto Oscillatory Neuronscited by 37
  19. 2022Ab-initio quantum chemistry with neural-network wavefunctionscited by 28
  20. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  21. 2022Fine-Tuning Pre-trained Transformers into Decaying Fast Weightscited by 26
  22. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 26
  23. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  24. 2024SpiNNaker2: A Large-Scale Neuromorphic System for Event-Based and Asynchronous Machine Learningcited by 24
  25. 2024Scalable MatMul-free Language Modelingcited by 21
  26. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 21
  27. 2023A Material Lens on Coloniality in NLPcited by 19
  28. 2021Anytime Dense Prediction with Confidence Adaptivitycited by 18
  29. 2023Unit Scaling: Out-of-the-Box Low-Precision Trainingcited by 17
  30. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  31. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  32. 2023Thermodynamic Computing System for AI Applicationscited by 16
  33. 2024KLay: Accelerating Arithmetic Circuits for Neurosymbolic AIcited by 16
  34. 2025Call for Action: towards the next generation of symbolic regression benchmarkcited by 16
  35. 2021Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networkscited by 15
  36. 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
  37. 2023Thermodynamic AI and the fluctuation frontiercited by 15
  38. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13

50 most cited of 160 All 160 (JSON)