mlsanityData to 7 Oct 2026

CodeIt: Self-Improving Language Models with Prioritized Hindsight Replay

arXiv 2402.04858 ↗First posted 7 Feb 2024cs.AI cs.CL cs.LGcited by 23

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large language models are increasingly solving tasks that are commonly believed to require human-level reasoning ability. However, these models still perform very poorly on benchmarks of general intelligence such as the Abstraction and Reasoning Corpus (ARC). In this paper, we approach ARC as a programming-by-examples problem, and introduce a novel and scalable method for language model self-improvement called Code Iteration (CodeIt). Our method iterates between 1) program sampling and hindsight relabeling, and 2) learning from prioritized experience replay. By relabeling the goal of an episode (i.e., the target program output given input) to the realized output produced by the sampled program, our method effectively deals with the extreme sparsity of rewards in program synthesis. Applying CodeIt to the ARC dataset, we demonstrate that prioritized hindsight replay, along with pre-training and data-augmentation, leads to successful inter-task generalization. CodeIt is the first neuro-symbolic approach that scales to the full ARC evaluation dataset. Our method solves 15% of ARC evaluation tasks, achieving state-of-the-art performance and outperforming existing neural and symbolic baselines. Our code is available at https://github.com/Qualcomm-AI-research/codeit .

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 52 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

33 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations26 May

Other references · 32

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2017Hindsight Experience Replaycited by 1,096
  3. 2015Prioritized Experience Replaycited by 1,070
  4. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  5. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 551
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  7. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  9. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
  10. 2023Large Language Models as General Pattern Machinescited by 196
  11. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  12. 2016DeepCoder: Learning to Write Programscited by 145
20 more
  1. 2022Formal Mathematics Statement Curriculum Learningcited by 141
  2. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  3. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
  4. 2023The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domaincited by 97
  5. 2016Neuro-Symbolic Program Synthesiscited by 84
  6. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  7. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  8. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  9. 2020Learning to Prove Theorems by Learning to Generate Theoremscited by 43
  10. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  11. 2021Fast and flexible: Human program induction in abstract reasoning taskscited by 33
  12. 2018Automatic Program Synthesis of Long Programs with a Learned Garbage Collectorcited by 24
  13. 2021Neural-guided, Bidirectional Program Search for Abstraction and Reasoningcited by 23
  14. 2021First Steps of an Approach to the ARC Challenge based on Descriptive Grid Models and the Minimum Description Length Principlecited by 13
  15. 2021Proving Theorems using Incremental Learning and Hindsight Experience Replaycited by 13
  16. 2023Unraveling the ARC Puzzle: Mimicking Human Solutions with Object-Centric Decision Transformercited by 12
  17. 2018Neural Program Search: Solving Programming Tasks from Description and Examplescited by 10
  18. 2022Learning Program Synthesis for Integer Sequences from Scratchcited by 8
  19. 2020Neural Abstract Reasonercited by 7
  20. 2023Tackling the Abstraction and Reasoning Corpus (ARC) with Object-centric Models and the MDL Principlecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 6; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 88
  2. 2024Combining Induction and Transduction for Abstract Reasoningcited by 53
  3. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 34
  4. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  5. 2024SELF-[IN]CORRECT: LLMs Struggle with Discriminating Self-Generated Responsescited by 24
  6. 2024Synthesizing Programmatic Reinforcement Learning Policies with Large Language Model Guided Searchcited by 13
  7. 2024Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcementcited by 12
  8. 2025Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewritingcited by 10
  9. 2024Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objectscited by 9
  10. 2024From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Modelscited by 8
  11. 2024ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)cited by 8
  12. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 8
11 more
  1. 2024ARCLE: The Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learningcited by 7
  2. 2025Don't throw the baby out with the bathwater: How and why deep learning for ARCcited by 7
  3. 2025Think Visually, Reason Textually: Vision-Language Synergy in ARCcited by 6
  4. 2024Program Synthesis using Inductive Logic Programming for the Abstraction and Reasoning Corpuscited by 4
  5. 2024Enhancing Analogical Reasoning in the Abstraction and Reasoning Corpus via Model-Based RLcited by 2
  6. 2025NSA: Neuro-symbolic ARC Challengecited by 2
  7. 2026Internalizing Agency from Reflective Experiencecited by 1
  8. 2026Unlocking LLM Code Correction with Iterative Feedback Loopscited by 1
  9. 2024System 2 Reasoning for Human-AI Alignment: Generality and Adaptivity via ARC-AGIcited by 0
  10. 2026JaxARC: A High-Performance JAX-based Environment for Abstraction and Reasoning Researchcited by 0
  11. 2026Quo Vadis, World Modeling?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Systematic Study of Small Language Models on Abstract Reasoning Tasks
  2. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  3. 2026ARC-TGI: Human-Validated Task Generators with Reasoning Chain Templates for ARC-AGI
  4. 2026DiARC: Distinguishing Positive and Negative Samples Helps Improving ARC-like Reasoning Ability of Large Language Models
  5. 2024Self-Improvement in Language Models: The Sharpening Mechanism
  6. 2025Tiny Recursive Models on ARC-AGI-1: Inductive Biases, Identity Conditioning, and Test-Time Compute
  7. 2026Pioneer Agent: Continual Improvement of Small Language Models in Production
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions