mlsanityData to 7 Oct 2026

Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language Models

arXiv 2305.18507 ↗First posted 29 May 2023cs.CL cs.AIcited by 16

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Large language models (LLMs) have scaled up to unlock a wide range of complex reasoning tasks with the aid of various prompting methods. However, current prompting methods generate natural language intermediate steps to help reasoning, which can cause imperfect task reduction and confusion. To mitigate such limitations, we explore code prompting, a neural symbolic prompting method with both zero-shot and few-shot versions which triggers code as intermediate steps. We conduct experiments on 7 widely-used benchmarks involving symbolic reasoning and arithmetic reasoning. Code prompting generally outperforms chain-of-thought (CoT) prompting. To further understand the performance and limitations of code prompting, we perform extensive ablation studies and error analyses, and identify several exclusive advantages of using symbolic promptings compared to natural language. We also consider the ensemble of code prompting and CoT prompting to combine the strengths of both. Finally, we show through experiments how code annotations and their locations affect code prompting.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 42 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

28 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks22 Nov
  2. 2022Automatic Chain of Thought Prompting in Large Language Models7 Oct
  3. 2022Complexity-Based Prompting for Multi-Step Reasoning3 Oct
  4. 2022Large Language Models are Zero-Shot Reasoners24 May
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Models21 May
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Models21 Mar
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  10. 2023Teaching Large Language Models to Self-Debugcited by 963
  11. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  12. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
9 more
  1. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  2. 2022PAL: Program-aided Language Modelscited by 635
  3. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  4. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  5. 2016Solving General Arithmetic Word Problemscited by 353
  6. 2022Binding Language Models in Symbolic Languagescited by 224
  7. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  8. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  9. 2018Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problemscited by 49

Built on by

Cited by year

Citing papers per year: 2023: 7; 2024: 3; 2025: 6.

16 papers.

Sort:

By area:

  1. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  2. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  3. 2024Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thoughtcited by 56
  4. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  5. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  6. 2023When Do Program-of-Thoughts Work for Reasoning?cited by 19
  7. 2023Mixed Distillation Helps Smaller Language Model Better Reasoningcited by 18
  8. 2025SymRTLO: Enhancing RTL Code Optimization with LLMs and Neuron-Inspired Symbolic Reasoningcited by 16
  9. 2023Natural Language Embedded Programs for Hybrid Language Symbolic Reasoningcited by 15
  10. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  11. 2023Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Modelscited by 10
  12. 2023Conditions for Length Generalization in Learning Reasoning Skillscited by 7
4 more
  1. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMscited by 7
  2. 2025Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendmentcited by 6
  3. 2025FinCoT: Grounding Chain-of-Thought in Expert Financial Reasoningcited by 5
  4. 2025RBF++: Quantifying and Optimizing Reasoning Boundaries across Measurable and Unmeasurable Capabilities for Chain-of-Thought Reasoningcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating Symbolic Capabilities of Large Language Models
  2. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMs
  3. 2023Large Language Models as Analogical Reasoners
  4. 2024Large Language Models are Contrastive Reasoners
  5. 2023Empirical Investigation of Neural Symbolic Reasoning Strategies
  6. 2024Towards Generalist Prompting for Large Language Models by Mental Models
  7. 2023Reasoning in Large Language Models Through Symbolic Math Word Problems
  8. 2026Evaluating Prompting and Execution-Based Methods for Deterministic Computation in LLMs