mlsanityData to 14 Oct 2026

Is Code Better Than Language for Algorithmic Reasoning

arXiv 2606.15589 ↗First posted 14 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

For tool-augmented language models, comparing natural-language reasoning with code-execution pipelines is difficult because the comparison changes both the intermediate representation and the execution mechanism. We separate these factors with an intermediate intervention: the model expresses its reasoning as executable code, and the language model simulates that code in context to produce an answer. On a 40-task verifiable algorithmic benchmark, deterministic code execution outperforms natural-language reasoning by +31.6pp. We observe that the intermediate intervention is not meaningfully different from natural-language reasoning (+0.15pp). These results suggest that, in our evaluated setting, changing the intermediate representation alone does not explain the tool-use advantage, providing evidence for the performance gains requiring reliable external execution. We formalize this intuition with a simple statistical decision-theoretic model that characterizes when execution dominates end-to-end risk in our disentangled trace-generation/execution regime. We validate our theory using a reconstruction intervention that leverages a proxy language model to infer natural-language reasoning traces from code representations, recovering performance comparable to the original natural-language reasoning pipeline. All experiments are at https://github.com/TerryTong-Git/ToolProj.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 28 of its 55 references that are in the map. First posted on arXiv 14 Jun 2026.

Builds on

28 of 55 references.

References · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  5. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  6. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  7. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  8. 2014Neural Turing Machinescited by 644
  9. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  10. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  11. 2015Neural Module Networkscited by 376
  12. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
16 more
  1. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  2. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 235
  3. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  4. 2022TALM: Tool Augmented Language Modelscited by 197
  5. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  6. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Proverscited by 168
  7. 2016Language to Logical Form with Neural Attentioncited by 158
  8. 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 109
  9. 2015Neural Programmer-Interpreterscited by 101
  10. 2024LLM With Tools: A Surveycited by 61
  11. 2021Few-Shot Semantic Parsing with Language Models Trained On Codecited by 50
  12. 2023The Expressive Power of Transformers with Chain of Thoughtcited by 50
  13. 2020Neural Execution Engines: Learning to Execute Subroutinescited by 36
  14. 2022Towards Better Out-of-Distribution Generalization of Neural Algorithmic Reasoning Taskscited by 30
  15. 2019Integrating Learning and Reasoning with Deep Logic Modelscited by 21
  16. 2025CoT Information: Improved Sample Complexity under Chain-of-Thought Supervisioncited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Exploring Extrinsic and Intrinsic Properties for Effective Reasoning with Code Interpreter
  2. 2025From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models
  3. 2026CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
  4. 2024Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models
  5. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning
  6. 2025Chain of Execution Supervision Promotes General Reasoning in Large Language Models
  7. 2026Teaching Language Models to Think in Code
  8. 2025On Code-Induced Reasoning in LLMs