mlsanityData to 7 Oct 2026

The CLRS-Text Algorithmic Reasoning Language Benchmark

arXiv 2406.04229 ↗First posted 6 Jun 2024cs.LG cs.AI cs.CLcited by 28

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Most recent studies dedicated to reasoning focus on out-of-distribution performance on procedurally-generated synthetic benchmarks, bespoke-built to evaluate specific skills only. This trend makes results hard to transfer across publications, slowing down progress. Three years ago, a similar issue was identified and rectified in the field of neural algorithmic reasoning, with the advent of the CLRS benchmark. CLRS is a dataset generator comprising graph execution traces of classical algorithms from the Introduction to Algorithms textbook. Inspired by this, we propose CLRS-Text -- a textual version of these algorithmic traces. Out of the box, CLRS-Text is capable of procedurally generating trace data for thirty diverse, challenging algorithmic tasks across any desirable input distribution, while offering a standard pipeline in which any additional algorithmic tasks may be created in the benchmark. We fine-tune and evaluate various LMs as generalist executors on this benchmark, validating prior work and revealing a novel, interesting challenge for the LM reasoning community. Our code is available at https://github.com/google-deepmind/clrs/tree/master/clrs/_src/clrs_text.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 50 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

33 of 50 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  11. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  12. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
21 more
  1. 2022Neural Networks and the Chomsky Hierarchycited by 214
  2. 2022Exploring Length Generalization in Large Language Modelscited by 204
  3. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  4. 2019What Can Neural Networks Reason About?cited by 155
  5. 2019Neural Execution of Graph Algorithmscited by 118
  6. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  7. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  8. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  9. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  10. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  11. 2024Understanding Transformer Reasoning Capabilities via Graph Algorithmscited by 64
  12. 2022A Generalist Neural Algorithmic Learnercited by 58
  13. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  14. 2023Neural Algorithmic Reasoning with Causal Regularisationcited by 30
  15. 2023Neural Algorithmic Reasoning for Combinatorial Optimisationcited by 19
  16. 2023SALSA-CLRS: A Sparse and Scalable Benchmark for Algorithmic Reasoningcited by 11
  17. 2024Beyond Lines and Circles: Unveiling the Geometric Reasoning Gap in Large Language Modelscited by 11
  18. 2024On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methodscited by 10
  19. 2023Recursive Algorithmic Reasoningcited by 9
  20. 2024Benchmarking ChatGPT on Algorithmic Reasoningcited by 8
  21. 2023Latent Space Representations of Neural Algorithmic Reasonerscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 9.

28 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  2. 2025Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Searchcited by 53
  3. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 48
  4. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  5. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  6. 2024Prompting Strategies for Enabling Large Language Models to Infer Causation from Correlationcited by 7
  7. 2026The Illusion of Stochasticity in LLMscited by 7
  8. 2024Recurrent Aggregators in Neural Algorithmic Reasoningcited by 6
  9. 2026Understanding Dynamic Compute Allocation in Recurrent Transformerscited by 6
  10. 2024Are Large-Language Models Graph Algorithmic Reasoners?cited by 5
  11. 2025L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Executioncited by 5
  12. 2025Looking beyond the next tokencited by 5
16 more
  1. 2025Can LLMs Reason Structurally? Benchmarking via the Lens of Data Structurescited by 5
  2. 2025Chain of Execution Supervision Promotes General Reasoning in Large Language Modelscited by 4
  3. 2025Attend or Perish: Benchmarking Attention in Algorithmic Reasoningcited by 2
  4. 2025Matching Markets Meet LLMs: Algorithmic Reasoning with Ranked Preferencescited by 2
  5. 2025Generalizable Insights for Graph Transformers in Theory and Practicecited by 2
  6. 2025Beyond Interpolation: Extrapolative Reasoning with Reinforcement Learning and Graph Neural Networkscited by 1
  7. 2025Improving LLMs' Generalized Reasoning Abilities by Graph Problemscited by 1
  8. 2026AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?cited by 1
  9. 2026AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worldscited by 1
  10. 2025GDS Agent for Graph Algorithmic Reasoningcited by 0
  11. 2025Efficiently Learning Branching Networks for Multitask Algorithmic Reasoningcited by 0
  12. 2026UltRAG: a Universal Simple Scalable Recipe for Knowledge Graph RAGcited by 0
  13. 2026Closed-Loop Graph Algorithm Execution with Small Language Models: Step Accuracy and Rollout Reliabilitycited by 0
  14. 2026SPARK: Susceptibility-Guided Profiling and Steering of Latent Reasoning States in Large Language Modelscited by 0
  15. 2026Cross-Model Cross-Language AI Coding Agent Performance: Accuracy and Speed of Parallel CLRS Algorithmscited by 0
  16. 2026World-Time Compute with Verified Code World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024CLR-Bench: Evaluating Large Language Models in College-level Reasoning
  2. 2024Open-Book Neural Algorithmic Reasoning
  3. 2026The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning
  4. 2025SLR: Automated Synthesis for Scalable Logical Reasoning
  5. 2025Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation
  6. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
  7. 2026Evaluating the Formal Reasoning Capabilities of Large Language Models through Chomsky Hierarchy
  8. 2026Reasoning in a Combinatorial and Constrained World: Benchmarking LLMs on Natural-Language Combinatorial Optimization