mlsanityData to 7 Oct 2026

Language Models of Code are Few-Shot Commonsense Learners

arXiv 2210.07128 ↗First posted 13 Oct 2022cs.CL cs.LGcited by 160

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We address the general task of structured commonsense reasoning: given a natural language input, the goal is to generate a graph such as an event -- or a reasoning-graph. To employ large language models (LMs) for this task, existing approaches ``serialize'' the output graph as a flat list of nodes and edges. Although feasible, these serialized graphs strongly deviate from the natural language corpora that LMs were pre-trained on, hindering LMs from generating them correctly. In this paper, we show that when we instead frame structured commonsense reasoning tasks as code generation tasks, pre-trained LMs of code are better structured commonsense reasoners than LMs of natural language, even when the downstream task does not involve source code at all. We demonstrate our approach across three diverse structured commonsense reasoning tasks. In all these natural language tasks, we show that using our approach, a code generation LM (CODEX) outperforms natural-LMs that are fine-tuned on the target task (e.g., T5) and other strong LMs such as GPT-3 in the few-shot setting.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 26 of its 39 references that are in the map. First posted on arXiv 13 Oct 2022.

Builds on

26 of 39 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
14 more
  1. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  2. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  3. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  4. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  5. 2022Autoformalization with Large Language Modelscited by 252
  6. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  7. 2019WIQA: A dataset for "What if..." reasoning over procedural textcited by 71
  8. 2018Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehensioncited by 62
  9. 2021ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoningcited by 58
  10. 2021Think about it! Improving defeasible reasoning by first modeling the question scenariocited by 20
  11. 2020Neural Language Modeling for Contextualized Temporal Graph Generationcited by 13
  12. 2021Could you give me a hint? Generating inference graphs for defeasible reasoningcited by 10
  13. 2022Coalescing Global and Local Information for Procedural Text Understandingcited by 10
  14. 2021CURIE: An Iterative Querying Approach for Reasoning About Situationscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 59; 2024: 47; 2025: 33; 2026: 12.

160 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2022PAL: Program-aided Language Modelscited by 635
  4. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  7. 2023Augmented Language Models: a Surveycited by 398
  8. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  9. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  10. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  11. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  12. 2022Reasoning with Language Model Prompting: A Surveycited by 251
38 more
  1. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  2. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  3. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  4. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  5. 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 162
  6. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  7. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  8. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  9. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  10. 2024GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluationscited by 118
  11. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  12. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  13. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  14. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  15. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  16. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 67
  17. 2024CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Modelscited by 65
  18. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  19. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  20. 2024To Code, or Not To Code? Exploring Impact of Code in Pre-trainingcited by 47
  21. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  22. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 46
  23. 2023LaMPilot: An Open Benchmark Dataset for Autonomous Driving with Language Model Programscited by 45
  24. 2024LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generationcited by 45
  25. 2023Entity Tracking in Language Modelscited by 44
  26. 2024Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scalecited by 43
  27. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  28. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  29. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  30. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  31. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  32. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  33. 2023CodeKGC: Code Language Model for Generative Knowledge Graph Constructioncited by 28
  34. 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
  35. 2024Unlocking Reasoning Potential in Large Langauge Models by Scaling Code-form Planningcited by 24
  36. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  37. 2023Causal Reasoning of Entities and Events in Procedural Textscited by 23
  38. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Com2: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
  2. 2024MIDGARD: Self-Consistency Using Minimum Description Length for Structured Commonsense Reasoning
  3. 2024ACCORD: Closing the Commonsense Measurability Gap
  4. 2022Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach
  5. 2024Eliciting Better Multilingual Structured Reasoning from LLMs through Code
  6. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
  7. 2024Can LLMs perform structured graph reasoning?
  8. 2024The Mystery of Compositional Generalization in Graph-based Generative Commonsense Reasoning