# Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models

arXiv: https://arxiv.org/abs/2404.02575 · HTML: https://mlsanity.com/paper/2404.02575/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 20 of its 23 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models](https://mlsanity.com/paper/2305.04091/) (2023)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango](https://mlsanity.com/paper/2209.07686/) (2022)
- [Teaching Algorithmic Reasoning via In-context Learning](https://mlsanity.com/paper/2211.09066/) (2022)
- [Self-Discover: Large Language Models Self-Compose Reasoning Structures](https://mlsanity.com/paper/2402.03620/) (2024)
- [Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph](https://mlsanity.com/paper/2009.11692/) (2020)
- [Chain of Thought Prompting Elicits Knowledge Augmentation](https://mlsanity.com/paper/2307.01640/) (2023)

## Built on by (most cited first, 20 of 20)

- [To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning](https://mlsanity.com/paper/2409.12183/) (2024)
- [When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs](https://mlsanity.com/paper/2505.11423/) (2025)
- [Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoning](https://mlsanity.com/paper/2502.17407/) (2025)
- [Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach](https://mlsanity.com/paper/2406.16386/) (2024)
- [MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs](https://mlsanity.com/paper/2412.15310/) (2024)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [Meta-Designing Quantum Experiments with Language Models](https://mlsanity.com/paper/2406.02470/) (2024)
- [Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation](https://mlsanity.com/paper/2511.02303/) (2025)
- [Learning to Reason via Program Generation, Emulation, and Search](https://mlsanity.com/paper/2405.16337/) (2024)
- [CodeAgents: A Token-Efficient Framework for Codified Multi-Agent Reasoning in LLMs](https://mlsanity.com/paper/2507.03254/) (2025)
- [FLARE: Faithful Logic-Aided Reasoning and Exploration](https://mlsanity.com/paper/2410.11900/) (2024)
- [Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM](https://mlsanity.com/paper/2407.04067/) (2024)
- [Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models](https://mlsanity.com/paper/2506.06008/) (2025)
- [Watch Your Steps: Observable and Modular Chains of Thought](https://mlsanity.com/paper/2409.15359/) (2024)
- [Abductive Logical Rule Induction by Bridging Inductive Logic Programming and Multimodal Large Language Models](https://mlsanity.com/paper/2509.21874/) (2025)
- [Semi-structured LLM Reasoners Can Be Rigorously Audited](https://mlsanity.com/paper/2505.24217/) (2025)
- [Exploration of Fast-Slow Latent Recurrence for Train-Short, Test-Long Generalization](https://mlsanity.com/paper/2604.01577/) (2026)
- [Large Language Models Are Self-Taught Reasoners: Enhancing LLM Applications via Tailored Problem-Solving Demonstrations](https://mlsanity.com/paper/2408.12315/) (2024)
- [Shaping Schema via Language Representation as the Next Frontier for LLM Intelligence Expanding](https://mlsanity.com/paper/2605.09271/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)

## Abstract (arXiv)

Algorithmic reasoning refers to the ability to understand the complex patterns behind the problem and decompose them into a sequence of reasoning steps towards the solution. Such nature of algorithmic reasoning makes it a challenge for large language models (LLMs), even though they have demonstrated promising performance in other reasoning tasks. Within this context, some recent studies use programming languages (e.g., Python) to express the necessary logic for solving a given instance/question (e.g., Program-of-Thought) as inspired by their strict and precise syntaxes. However, it is non-trivial to write an executable code that expresses the correct logic on the fly within a single inference call. Also, the code generated specifically for an instance cannot be reused for others, even if they are from the same task and might require identical logic to solve. This paper presents Think-and-Execute, a novel framework that decomposes the reasoning process of language models into two steps. (1) In Think, we discover a task-level logic that is shared across all instances for solving a given task and then express the logic with pseudocode; (2) In Execute, we further tailor the generated pseudocode to each instance and simulate the execution of the code. With extensive experiments on seven algorithmic reasoning tasks, we demonstrate the effectiveness of Think-and-Execute. Our approach better improves LMs' reasoning compared to several strong baselines performing instance-specific reasoning (e.g., CoT and PoT), suggesting the helpfulness of discovering task-level logic. Also, we show that compared to natural language, pseudocode can better guide the reasoning of LMs, even though they are trained to follow natural language instructions.