mlsanityData to 14 Oct 2026

Natural Language Embedded Programs for Hybrid Language Symbolic Reasoning

arXiv 2309.10814 ↗First posted 19 Sep 2023cs.CLcited by 15

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

How can we perform computations over natural language representations to solve tasks that require symbolic and numeric reasoning? We propose natural language embedded programs (NLEP) as a unifying framework for addressing math/symbolic reasoning, natural language understanding, and instruction following tasks. Our approach prompts a language model to generate full Python programs that define functions over data structures which contain natural language representations of structured knowledge. A Python interpreter then executes the generated code and prints the output. Despite using a task-general prompt, we find that this approach can improve upon strong baselines across a range of different tasks including math and symbolic reasoning, text classification, question answering, and instruction following. We found that the generated programs are interpretable since they outline the exact reasoning process followed by the program interpreter.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 36 of its 49 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

36 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Code Llama: Open Foundation Models for Code24 Aug
  2. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance26 May
  3. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  4. 2023Scallop: A Language for Neurosymbolic Programming10 Apr
  5. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  6. 2023Augmented Language Models: a Survey15 Feb
  7. 2022PAL: Program-aided Language Models18 Nov
  8. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
16 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  2. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  4. 2018Neural Network Acceptability Judgmentscited by 958
  5. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  9. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  10. 2023Large Language Models as Tool Makerscited by 303
  11. 2022Binding Language Models in Symbolic Languagescited by 228
  12. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
  13. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 197
  14. 2023TaskMatrix.AI: Completing Tasks by Connecting Foundation Models with Millions of APIscited by 184
  15. 2023Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language Modelscited by 16
  16. 2023Entailment as Robust Self-Learnercited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 4; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  2. 2024THREAD: Thinking Deeper with Recursive Spawningcited by 25
  3. 2023Mixed Distillation Helps Smaller Language Model Better Reasoningcited by 19
  4. 2024Can We Verify Step by Step for Incorrect Answer Detection?cited by 15
  5. 2025PLAY2PROMPT: Zero-shot Tool Instruction Optimization for LLM Agents via Tool Playcited by 14
  6. 2023Towards Verifiable Text Generation with Symbolic Referencescited by 13
  7. 2024What Makes a Maze Look Like a Maze?cited by 11
  8. 2024Learning to Reason via Program Generation, Emulation, and Searchcited by 10
  9. 2024Cognitive Kernel: An Open-source Agent System towards Generalist Autopilotscited by 10
  10. 2024Neuro-Symbolic AI: Explainability, Challenges, and Future Trendscited by 10
  11. 2025Generate, Discriminate, Evolve: Enhancing Context Faithfulness via Fine-Grained Sentence-Level Self-Evolutioncited by 7
  12. 2026Beyond Single-Shot: Multi-step Tool Retrieval via Query Planningcited by 3
3 more
  1. 2024Bridging the Gap: Representation Spaces in Neuro-Symbolic AIcited by 0
  2. 2025Expertise-aware Multi-LLM Recruitment and Collaboration for Medical Decision-Makingcited by 0
  3. 2025LLM Enhancement with Domain Expert Mental Model to Reduce LLM Hallucination with Causal Prompt Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning
  2. 2026From Natural Language to Executable Narsese: A Neuro-Symbolic Benchmark and Pipeline for Reasoning with NARS
  3. 2023Reasoning in Large Language Models Through Symbolic Math Word Problems
  4. 2024HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning
  5. 2023Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models
  6. 2025From Language to Logic: A Bi-Level Framework for Structured Reasoning
  7. 2022Reasoning Like Program Executors
  8. 2026Moxia: A Trust-First Neuro-Symbolic Execution Architecture for Self-Explaining Mathematical Reasoning