mlsanityData to 7 Oct 2026

Benchmarking ChatGPT on Algorithmic Reasoning

arXiv 2404.03441 ↗First posted 4 Apr 2024cs.AI cs.CL cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We evaluate ChatGPT's ability to solve algorithm problems from the CLRS benchmark suite that is designed for GNNs. The benchmark requires the use of a specified classical algorithm to solve a given problem. We find that ChatGPT outperforms specialist GNN models, using Python to successfully solve these problems. This raises new points in the discussion about learning algorithms with neural networks and how we think about what out of distribution testing looks like with web scale training data.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

21 of 24 references.

References · 21

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2022PAL: Program-aided Language Modelscited by 635
  4. 2019Neural Execution of Graph Algorithmscited by 118
  5. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  6. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  7. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  8. 2022A Generalist Neural Algorithmic Learnercited by 58
  9. 2022Graph Neural Networks are Dynamic Programmerscited by 55
  10. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  11. 2023Neural Algorithmic Reasoning with Causal Regularisationcited by 30
  12. 2024Simulation of Graph Algorithms with Looped Transformerscited by 29
9 more
  1. 2023Dual Algorithmic Reasoningcited by 21
  2. 2023Neural Algorithmic Reasoning for Combinatorial Optimisationcited by 19
  3. 2023Neural Algorithmic Reasoning Without Intermediate Supervisioncited by 16
  4. 2023SALSA-CLRS: A Sparse and Scalable Benchmark for Algorithmic Reasoningcited by 11
  5. 2024On the Markov Property of Neural Algorithmic Reasoning: Analyses and Methodscited by 10
  6. 2023Recursive Algorithmic Reasoningcited by 9
  7. 2023Neural Priority Queues for Graph Neural Networkscited by 7
  8. 2023Latent Space Representations of Neural Algorithmic Reasonerscited by 6
  9. 2023Triplet Edge Attention for Algorithmic Reasoningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 2; 2026: 1.

8 papers.

  1. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  2. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 48
  3. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  4. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  5. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 13
  6. 2024Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Pathcited by 10
  7. 2024Are Large-Language Models Graph Algorithmic Reasoners?cited by 5
  8. 2026FloydNet: A Learning Paradigm for Global Relational Reasoningcited by 1