mlsanityData to 14 Oct 2026

The CLRS Algorithmic Reasoning Benchmark

arXiv 2205.15659 ↗First posted 31 May 2022cs.LG cs.DS stat.MLcited by 106

Not on the map. Why?

Abstract (arXiv)

Learning representations of algorithms is an emerging area of machine learning, seeking to bridge concepts from neural networks with classical algorithms. Several important works have investigated whether neural networks can effectively reason like algorithms, typically by learning to execute them. The common trend in the area, however, is to generate targeted kinds of algorithmic data to evaluate specific hypotheses, making results hard to transfer across publications, and increasing the barrier of entry. To consolidate progress and work towards unified evaluation, we propose the CLRS Algorithmic Reasoning Benchmark, covering classical algorithms from the Introduction to Algorithms textbook. Our benchmark spans a variety of algorithmic reasoning procedures, including sorting, searching, dynamic programming, graph algorithms, string algorithms and geometric algorithms. We perform extensive experiments to demonstrate how several popular algorithmic reasoning baselines perform on these tasks, and consequently, highlight links to several open challenges. Our library is readily available at https://github.com/deepmind/clrs.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Builds on 48 of its 84 references that are in the map. First posted on arXiv 31 May 2022.

Builds on

48 of 84 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017Graph Attention Networkscited by 8,923
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2017Deep Setscited by 1,421
  10. 2015Pointer Networkscited by 895
  11. 2021How Attentive are Graph Attention Networks?cited by 835
  12. 2020Benchmarking Graph Neural Networkscited by 680
36 more
  1. 2018Attention, Learn to Solve Routing Problems!cited by 656
  2. 2014Neural Turing Machinescited by 644
  3. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  4. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 623
  5. 2017Learning Combinatorial Optimization Algorithms over Graphscited by 561
  6. 2020Principal Neighbourhood Aggregation for Graph Netscited by 478
  7. 2021Combinatorial optimization and reasoning with graph neural networkscited by 268
  8. 2020Can Graph Neural Networks Count Substructures?cited by 236
  9. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  10. 2019Two models of double descent for weak featurescited by 195
  11. 2021PonderNet: Learning to Pondercited by 168
  12. 2019What Can Neural Networks Reason About?cited by 159
  13. 2015Neural GPUs Learn Algorithmscited by 121
  14. 2019Neural Execution of Graph Algorithmscited by 121
  15. 2014Learning to Executecited by 106
  16. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 100
  17. 2019Compositional generalization through meta sequence-to-sequence learningcited by 99
  18. 2021Neural Algorithmic Reasoningcited by 94
  19. 2021Size-Invariant Graph Representations for Graph Classification Extrapolationscited by 84
  20. 2018Neural Arithmetic Logic Unitscited by 70
  21. 2022Graph Neural Networks are Dynamic Programmerscited by 55
  22. 2020Pointer Graph Networkscited by 53
  23. 2018Modular meta-learningcited by 51
  24. 2018Relational inductive bias for physical construction in humans and machinescited by 44
  25. 2021Simple GNN Regularisation for 3D Molecular Property Prediction & Beyondcited by 44
  26. 2020Neural Execution Engines: Learning to Execute Subroutinescited by 36
  27. 2020It's Not What Machines Can Learn, It's What We Cannot Teachcited by 29
  28. 2021How to transfer algorithmic reasoning knowledge to learn new algorithms?cited by 22
  29. 2020MEMO: A Deep Network for Flexible Combination of Episodic Memoriescited by 21
  30. 2021Neural Algorithmic Reasoners are Implicit Plannerscited by 21
  31. 2020Normalized Attention Without Probability Cagecited by 20
  32. 2020Neural Bipartite Matchingcited by 17
  33. 2021Persistent Message Passingcited by 13
  34. 2021Reasoning-Modulated Representationscited by 13
  35. 2020Strong Generalization and Efficiency in Neural Programscited by 12
  36. 2020Graph neural induction of value iterationcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 26; 2024: 30; 2025: 21; 2026: 22.

106 papers.

Sort:

By area:

  1. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  2. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 168
  3. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 131
  4. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 124
  5. 2023LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimizationcited by 87
  6. 2024Position: Graph Foundation Models are Already Herecited by 86
  7. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  8. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  9. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  10. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 67
  11. 2022A Generalist Neural Algorithmic Learnercited by 59
  12. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 53
38 more
  1. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 53
  2. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 49
  3. 2023TacticAI: an AI assistant for football tacticscited by 44
  4. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  5. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  6. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 34
  7. 2023Neural Algorithmic Reasoning with Causal Regularisationcited by 31
  8. 2022Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Taskscited by 30
  9. 2024Simulation of Graph Algorithms with Looped Transformerscited by 30
  10. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  11. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  12. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  13. 2023Dual Algorithmic Reasoningcited by 21
  14. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  15. 2023Neural Algorithmic Reasoning for Combinatorial Optimisationcited by 19
  16. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  17. 2023Neural Algorithmic Reasoning Without Intermediate Supervisioncited by 17
  18. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 16
  19. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  20. 2023Asynchronous Algorithmic Alignment with Cocyclescited by 13
  21. 2024PUZZLES: A Benchmark for Neural Algorithmic Reasoningcited by 13
  22. 2023SALSA-CLRS: A Sparse and Scalable Benchmark for Algorithmic Reasoningcited by 12
  23. 2023Are Graph Neural Networks Optimal Approximation Algorithms?cited by 11
  24. 2024Towards Principled Graph Transformerscited by 11
  25. 2024On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methodscited by 11
  26. 2025Graph neural networks extrapolate out-of-distribution for shortest pathscited by 11
  27. 2023Recursive Algorithmic Reasoningcited by 9
  28. 2024Discrete Neural Algorithmic Reasoningcited by 9
  29. 2025IOLBENCH: Benchmarking LLMs on Linguistic Reasoningcited by 9
  30. 2025Primal-Dual Neural Algorithmic Reasoningcited by 9
  31. 2024Benchmarking ChatGPT on Algorithmic Reasoningcited by 8
  32. 2023Latent Space Representations of Neural Algorithmic Reasonerscited by 7
  33. 2023Neural Priority Queues for Graph Neural Networkscited by 7
  34. 2024Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlationcited by 7
  35. 2022Learning Graph Algorithms With Recurrent Graph Neural Networkscited by 6
  36. 2023Parallel Algorithms Align with Neural Executioncited by 6
  37. 2024Future Directions in the Theory of Graph Machine Learningcited by 6
  38. 2024Recurrent Aggregators in Neural Algorithmic Reasoningcited by 6

50 most cited of 106 All 106 (JSON)