mlsanityData to 7 Oct 2026

Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation

arXiv 2310.02304 ↗First posted 3 Oct 2023cs.CL cs.AI cs.LGcited by 128

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Several recent advances in AI systems solve problems by providing a "scaffolding" program that structures multiple calls to language models (LMs) to generate better outputs. A scaffolding program is written in a programming language such as Python. In this work, we use a language-model-infused scaffolding program to improve itself. We start with a seed "improver" that improves an input program according to a given utility function by querying an LM several times and returning the best solution. We then run this seed improver to improve itself. Across a small set of downstream tasks, the resulting improved improver generates programs with significantly better performance than its seed improver. A variety of self-improvement strategies are proposed by the language model, including beam search, genetic algorithms, and simulated annealing. Since the language models themselves are not altered, this is not full recursive self-improvement. Nonetheless, it demonstrates that a modern language model, GPT-4 in our experiments, is capable of writing code that can call itself to improve itself. We consider concerns around the development of self-improving technologies and evaluate the frequency with which the generated code bypasses a sandbox.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 54 of its 90 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

54 of 90 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution28 Sep

Other references · 53

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  5. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  9. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  12. 2023Large Language Models as Optimizerscited by 830
  13. 2022Large Language Models Can Self-Improvecited by 656
  14. 2022PAL: Program-aided Language Modelscited by 635
  15. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  16. 2023Cognitive Architectures for Language Agentscited by 422
  17. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  18. 2022CodeT: Code Generation with Generated Testscited by 369
  19. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  20. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  21. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  22. 2023Large Language Models as Tool Makerscited by 293
  23. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  24. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  25. 2022Predictability and Surprise in Large Generative Modelscited by 209
  26. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  27. 2023Planning with Large Language Models for Code Generationcited by 179
  28. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
  29. 2022Language Model Cascadescited by 104
  30. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  31. 2023Learning Performance-Improving Code Editscited by 103
  32. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  33. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  34. 2022Parsel: Algorithmic Reasoning with Language Models by Composing Decompositionscited by 62
  35. 2022Top-Down Synthesis for Library Learningcited by 57
  36. 2015Program Synthesis using Natural Languagecited by 41
  37. 2019Unsupervised Paraphrasing by Simulated Annealingcited by 37
  38. 2021The Challenge of Value Alignment: from Fairer Algorithms to AI Safetycited by 34

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 28; 2025: 25; 2026: 73.

128 papers.

Sort:

By area:

  1. 2024Automated Design of Agentic Systemscited by 360
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 292
  3. 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
  4. 2024Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systemscited by 129
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  7. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  8. 2024G-Designer: Architecting Multi-agent Communication Topologies via Graph Neural Networkscited by 110
  9. 2024LLaMEA: A Large Language Model Evolutionary Algorithm for Automatically Generating Metaheuristicscited by 92
  10. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  11. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 86
  12. 2024Language Agents as Optimizable Graphscited by 81
38 more
  1. 2026Self-Harness: Harnesses That Improve Themselvescited by 79
  2. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  3. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  4. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 69
  5. 2024StateFlow: Enhancing LLM Task-Solving through State-Driven Workflowscited by 65
  6. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  7. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  8. 2025A Self-Improving Coding Agentcited by 58
  9. 2023Prompt Engineering a Prompt Engineercited by 53
  10. 2025Huxley-G"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machinecited by 44
  11. 2024Evolving Code with A Large Language Modelcited by 42
  12. 2026Group-Evolving Agents: Open-Ended Self-Improvement via Experience Sharingcited by 32
  13. 2024G"odel Agent: A Self-Referential Agent Framework for Recursive Self-Improvementcited by 27
  14. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 26
  15. 2024Understanding the Importance of Evolutionary Search in Automated Heuristic Design with Large Language Modelscited by 22
  16. 2026Recursive Harness Self-Improvementcited by 22
  17. 2025Learning to Reason via Mixture-of-Thought for Logical Reasoningcited by 20
  18. 2026The Red Queen G"odel Machine: Co-Evolving Agents and Their Evaluatorscited by 16
  19. 2025Beyond Prompt Content: Enhancing LLM Performance via Content-Format Integrated Prompt Optimizationcited by 15
  20. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  21. 2024Agent Alignment in Evolving Social Normscited by 14
  22. 2024Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMscited by 13
  23. 2025Self-Steering Language Modelscited by 13
  24. 2026VeRO: A Harness for Agents to Optimize Agentscited by 12
  25. 2026MetaSkill-Evolve: Recursive Self-Improvement of LLM Agents via Two-Timescale Meta-Skill Evolutioncited by 12
  26. 2026TTHE: Test-Time Harness Evolutioncited by 12
  27. 2024Evaluating and Optimizing Educational Content with Large Language Model Judgmentscited by 11
  28. 2025Feedback Friction: LLMs Struggle to Fully Incorporate External Feedbackcited by 11
  29. 2024I am a Strange Dataset: Metalinguistic Tests for Language Modelscited by 10
  30. 2025CMCTS: A Constrained Monte Carlo Tree Search Framework for Mathematical Reasoning in Large Language Modelcited by 9
  31. 2024Introducing MAPO: Momentum-Aided Gradient Descent Prompt Optimizationcited by 8
  32. 2026Evo-Bench: Can Language Models Improve Agent Harness?cited by 8
  33. 2025Generalizable Heuristic Generation Through LLMs with Meta-Optimizationcited by 7
  34. 2026Can AI agents conduct open-ended AI research? Early evidence from two case studiescited by 7
  35. 2025AgentDropout: Dynamic Agent Elimination for Token-Efficient and High-Performance LLM-Based Multi-Agent Collaborationcited by 6
  36. 2024tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs)cited by 5
  37. 2025Inefficiencies of Meta Agents for Agent Designcited by 5
  38. 2026HarnessOpt-Bench: Evaluating LLMs at Harness Optimizationcited by 5

50 most cited of 128 All 128 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Improvement in Language Models: The Sharpening Mechanism
  2. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  3. 2026ImproveAnyTask: An Autonomous Post-Training Harness for Iterative Model Self-Improvement
  4. 2025Self-Correcting Code Generation Using Small Language Models
  5. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
  6. 2026Learning to Optimize through Solver-Grounded Self-Play
  7. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  8. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generation