mlsanityData to 7 Oct 2026

Reasoning Like Program Executors

arXiv 2201.11473 ↗First posted 27 Jan 2022cs.CL cs.AI cs.SCcited by 45

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Reasoning over natural language is a long-standing goal for the research community. However, studies have shown that existing language models are inadequate in reasoning. To address the issue, we present POET, a novel reasoning pre-training paradigm. Through pre-training language models with programs and their execution results, POET empowers language models to harvest the reasoning knowledge possessed by program executors via a data-driven approach. POET is conceptually simple and can be instantiated by different kinds of program executors. In this paper, we showcase two simple instances POET-Math and POET-Logic, in addition to a complex instance, POET-SQL. Experimental results on six benchmarks demonstrate that POET can significantly boost model performance in natural language reasoning, such as numerical reasoning, logical reasoning, and multi-hop reasoning. POET opens a new gate on reasoning-enhancement pre-training, and we hope our analysis would shed light on the future research of reasoning like program executors.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 58 of its 86 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

58 of 86 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
38 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  5. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  6. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  7. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  8. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  9. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 402
  10. 2015Neural Module Networkscited by 375
  11. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  12. 2019Abductive Commonsense Reasoningcited by 300
  13. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  14. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 233
  15. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  16. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  17. 2017Neural-Symbolic Learning and Reasoning: A Survey and Interpretationcited by 195
  18. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  19. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  20. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  21. 2019Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoningcited by 100
  22. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  23. 2019Hierarchical Graph Network for Multi-hop Question Answeringcited by 83
  24. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 77
  25. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  26. 2020UnNatural Language Inferencecited by 68
  27. 2019Cognitive Graph for Multi-Hop Reading Comprehension at Scalecited by 65
  28. 2019Neural Module Networks for Reasoning over Textcited by 65
  29. 2020Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Trackingcited by 65
  30. 2020Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Modelscited by 65
  31. 2019Dynamically Fused Graph Network for Multi-hop Reasoningcited by 63
  32. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  33. 2020On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queriescited by 59
  34. 2019Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documentscited by 57
  35. 2018Robust Text-to-SQL Generation with Execution-Guided Decodingcited by 51
  36. 2019Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehensioncited by 51
  37. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  38. 2019A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoningcited by 41

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 17; 2023: 22; 2024: 4; 2025: 2.

45 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2022PAL: Program-aided Language Modelscited by 635
  5. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  6. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  7. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  8. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  9. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 132
  10. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  11. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  12. 2023Natural Language Reasoning, A Surveycited by 61
33 more
  1. 2024NExT: Teaching Large Language Models to Reason about Code Executioncited by 60
  2. 2023Generative Table Pre-training Empowers Models for Tabular Predictioncited by 55
  3. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  4. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  5. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  6. 2023Learning Deep Semantics for Test Completioncited by 31
  7. 2022ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examplescited by 29
  8. 2023CodeCoT: Tackling Code Syntax Errors in CoT Reasoning for Code Generationcited by 29
  9. 2023Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarioscited by 18
  10. 2023Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Modelscited by 16
  11. 2023From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuningcited by 14
  12. 2023Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networkscited by 12
  13. 2023Learning Multi-Step Reasoning by Solving Arithmetic Taskscited by 12
  14. 2022LogiGAN: Learning Logical Reasoning via Adversarial Pre-trainingcited by 11
  15. 2023SEER : A Knapsack approach to Exemplar Selection for In-Context HybridQAcited by 9
  16. 2022Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contextscited by 7
  17. 2022Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systemscited by 7
  18. 2022A Survey on Table-and-Text HybridQA: Concepts, Methods, Challenges and Future Directionscited by 7
  19. 2023Math Word Problem Solving by Generating Linguistic Variants of Problem Statementscited by 7
  20. 2023Self-Convinced Prompting: Few-Shot Question Answering with Repeated Introspectioncited by 7
  21. 2022Towards Robust Numerical Question Answering: Diagnosing Numerical Capabilities of NLP Systemscited by 6
  22. 2022Optimization Techniques for Unsupervised Complex Table Reasoning via Self-Training Frameworkcited by 6
  23. 2022NAPG: Non-Autoregressive Program Generation for Hybrid Tabular-Textual Question Answeringcited by 4
  24. 2023Scaling Experiments in Self-Supervised Cross-Table Representation Learningcited by 4
  25. 2024Improving Language Model Reasoning with Self-motivated Learningcited by 4
  26. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questionscited by 2
  27. 2023Pento-DIARef: A Diagnostic Dataset for Learning the Incremental Algorithm for Referring Expression Generation from Examplescited by 1
  28. 2023FootGPT : A Large Language Model Development Experiment on a Minimal Settingcited by 1
  29. 2023Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoningcited by 0
  30. 2023Multi-Method Self-Training: Improving Code Generation With Text, And Vice Versacited by 0
  31. 2024Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solvingcited by 0
  32. 2024Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)cited by 0
  33. 2025A Diversity-Enhanced Knowledge Distillation Model for Practical Math Word Problem Solvingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LogicAsker: Evaluating and Improving the Logical Reasoning Ability of Large Language Models
  2. 2026Is Code Better Than Language for Algorithmic Reasoning
  3. 2025Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment
  4. 2024Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models
  5. 2025Chain of Execution Supervision Promotes General Reasoning in Large Language Models
  6. 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation
  7. 2025L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution
  8. 2025Understanding Benchmark Language Under Weakened Formal Semantics