mlsanityData to 7 Oct 2026

Hypothesis Search: Inductive Reasoning with Language Models

arXiv 2309.05660 ↗First posted 11 Sep 2023cs.LG cs.AI cs.CLcited by 119

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Inductive reasoning is a core problem-solving capacity: humans can identify underlying principles from a few examples, which robustly generalize to novel scenarios. Recent work evaluates large language models (LLMs) on inductive reasoning tasks by directly prompting them yielding "in context learning." This works well for straightforward inductive tasks but performs poorly on complex tasks such as the Abstraction and Reasoning Corpus (ARC). In this work, we propose to improve the inductive reasoning ability of LLMs by generating explicit hypotheses at multiple levels of abstraction: we prompt the LLM to propose multiple abstract hypotheses about the problem, in natural language, then implement the natural language hypotheses as concrete Python programs. These programs can be verified by running on observed examples and generalized to novel inputs. To reduce the hypothesis search space, we explore steps to filter the set of hypotheses to implement: we either ask the LLM to summarize them into a smaller set of hypotheses or ask human annotators to select a subset. We verify our pipeline's effectiveness on the ARC visual inductive reasoning benchmark, its variant 1D-ARC, string transformation dataset SyGuS, and list transformation dataset List Functions. On a random 100-problem subset of ARC, our automated pipeline using LLM summaries achieves 30% accuracy, outperforming the direct prompting baseline (accuracy of 17%). With the minimal human input of selecting from LLM-generated candidates, performance is boosted to 33%. Our ablations show that both abstract hypothesis generation and concrete program representations benefit LLMs on inductive reasoning tasks.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 41 of its 62 references that are in the map. First posted on arXiv 11 Sep 2023.

Builds on

41 of 62 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions20 Dec
  2. 2022Emergent Analogical Reasoning in Large Language Models19 Dec
  3. 2022PAL: Program-aided Language Models18 Nov
  4. 2022Language models show human-like content effects on reasoning tasks14 Jul
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 36

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  10. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  11. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 369
  12. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
24 more
  1. 2022Can language models learn from explanations in context?cited by 248
  2. 2023Large Language Models as General Pattern Machinescited by 196
  3. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  4. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  5. 2016Language to Logical Form with Neural Attentioncited by 158
  6. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  7. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  8. 2017CSGNet: Neural Shape Parser for Constructive Solid Geometrycited by 106
  9. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  10. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  11. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  12. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  13. 2022Coder Reviewer Reranking for Code Generationcited by 83
  14. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 81
  15. 2017Learning with Latent Languagecited by 75
  16. 2015Program Synthesis using Natural Languagecited by 41
  17. 2017From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihoodcited by 41
  18. 2019Neural Program Repair by Jointly Learning to Localize and Repaircited by 39
  19. 2021Communicating Natural Programs to Humans and Machinescited by 35
  20. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33
  21. 2022Graphs, Constraints, and Search for the Abstraction and Reasoning Corpuscited by 32
  22. 2020BUSTLE: Bottom-Up Program Synthesis Through Learning-Guided Explorationcited by 30
  23. 2022CrossBeam: Learning to Search in Bottom-Up Program Synthesiscited by 18
  24. 2017Interactive Program Synthesiscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 46; 2025: 50; 2026: 17.

119 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  3. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  4. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  5. 2024LLM-SR: Scientific Equation Discovery via Programming with Large Language Modelscited by 117
  6. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  7. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  8. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 88
  9. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  10. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  11. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 54
  12. 2024Combining Induction and Transduction for Abstract Reasoningcited by 53
38 more
  1. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 50
  2. 2025Wider or Deeper? Scaling LLM Inference-Time Compute with Adaptive Branching Tree Searchcited by 48
  3. 2024Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Modelscited by 45
  4. 2024Automated Statistical Model Discovery with Language Modelscited by 41
  5. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 40
  6. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 34
  7. 2024Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpuscited by 32
  8. 2024H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmarkcited by 32
  9. 2025Curie: Toward Rigorous and Automated Scientific Experimentation with AI Agentscited by 31
  10. 2025The Curse of CoT: On the Limitations of Chain-of-Thought in In-Context Learningcited by 30
  11. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  12. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  13. 2024Is Programming by Example solved by LLMs?cited by 27
  14. 2024EEG-GPT: Exploring Capabilities of Large Language Models for EEG Classification and Interpretationcited by 26
  15. 2025Thinking Machines: A Survey of LLM based Reasoning Strategiescited by 25
  16. 2025PoE-World: Compositional World Modeling with Products of Programmatic Expertscited by 25
  17. 2024CodeIt: Self-Improving Language Models with Prioritized Hindsight Replaycited by 23
  18. 2024Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoningcited by 23
  19. 2024Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmarkcited by 22
  20. 2025ArcMemo: Abstract Reasoning Composition with Lifelong LLM Memorycited by 22
  21. 2024IDEA: Enhancing the Rule Learning Ability of Large Language Model Agent through Induction, Deduction, and Abductioncited by 20
  22. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answeringcited by 17
  23. 2024IW-Bench: Evaluating Large Multimodal Models for Converting Image-to-Webcited by 16
  24. 2025CER: Confidence Enhanced Reasoning in LLMscited by 16
  25. 2025BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discoverycited by 15
  26. 2024HYSYNTH: Context-Free LLM Approximation for Guiding Program Synthesiscited by 14
  27. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  28. 2026LoopViT: Scaling Visual ARC with Looped Transformerscited by 14
  29. 2023Explain-then-Translate: An Analysis on Improving Program Translation with Self-generated Explanationscited by 12
  30. 2024Loose LIPS Sink Ships: Asking Questions in Battleship with Language-Informed Program Samplingcited by 12
  31. 2024CausalGraph2LLM: Evaluating LLMs for Causal Queriescited by 12
  32. 2023Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solvingcited by 11
  33. 2024The Role of Foundation Models in Neuro-Symbolic Learning and Reasoningcited by 11
  34. 2024Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoningcited by 11
  35. 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
  36. 2024Mars: Situated Inductive Reasoning in an Open-World Environmentcited by 10
  37. 2025WorldLLM: Improving LLMs' world modeling using curiosity-driven theory-makingcited by 10
  38. 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 10

50 most cited of 119 All 119 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Inductive Reasoning for Large Language Models
  2. 2024MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models
  3. 2024Generating Diverse Hypotheses for Inductive Reasoning
  4. 2024The Role of Deductive and Inductive Reasoning in Large Language Models
  5. 2025Code-Driven Inductive Synthesis: Enhancing Reasoning Abilities of Large Language Models with Sequences
  6. 2024Conceptual and Unbiased Reasoning in Language Models
  7. 2025Code-driven Number Sequence Calculation: Enhancing the inductive Reasoning Abilities of Large Language Models
  8. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance