mlsanityData to 7 Oct 2026

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

arXiv 2207.01780 ↗First posted 5 Jul 2022cs.LG cs.CL cs.PLcited by 342

Not on the map. Why?

Cited by 58 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train a code generation model only from the pairs of natural-language problem descriptions and ground-truth programs. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus often results in poor performance when solving complex unseen coding tasks. To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL). Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives, larger model sizes, and better pretraining data. Our method not only achieves new SOTA results on the challenging APPS benchmark, but also shows strong zero-shot transfer capability with new SOTA results on the simpler MBPP benchmark.

Key facts. Cited by 342 papers in the map (data through 7 Oct 2026). Builds on 50 of its 92 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

50 of 92 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2015Continuous control with deep reinforcement learningcited by 3,279
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
38 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 879
  2. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  5. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  6. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  7. 2016Self-critical Sequence Training for Image Captioningcited by 544
  8. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  9. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  10. 2020IntelliCode Compose: Code Generation Using Transformercited by 225
  11. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  12. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  13. 2022Synchromesh: Reliable code generation from pre-trained language modelscited by 156
  14. 2016DeepCoder: Learning to Write Programscited by 145
  15. 2017SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learningcited by 135
  16. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  17. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  18. 2015Deep Convolutional Inverse Graphics Networkcited by 121
  19. 2019Write, Execute, Assess: Program Synthesis with a REPLcited by 92
  20. 2016Neuro-Symbolic Program Synthesiscited by 84
  21. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  22. 2020PyMT5: multi-mode translation of natural language and Python code with transformerscited by 80
  23. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  24. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  25. 2018Synthesizing Programs for Images using Reinforced Adversarial Learningcited by 72
  26. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  27. 2021Learning to Synthesize Programs as Interpretable and Generalizable Policiescited by 59
  28. 2018Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsingcited by 57
  29. 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
  30. 2017From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihoodcited by 41
  31. 2017Deep Reinforcement Learning-based Image Captioning with Embedding Rewardcited by 39
  32. 2020Fast and Memory-Efficient Neural Code Completioncited by 38
  33. 2021Latent Execution for Neural Program Synthesiscited by 35
  34. 2015Neural Random-Access Machinescited by 32
  35. 2016Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesiscited by 24
  36. 2021Learning to Complete Code with Sketchescited by 23
  37. 2020Learning Autocompletion from Real-World Datasetscited by 19
  38. 2021VarCLR: Variable Semantic Representation Pre-training via Contrastive Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 63; 2024: 70; 2025: 93; 2026: 109.

342 papers.

Sort:

By area:

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  9. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  10. 2023Cognitive Architectures for Language Agentscited by 422
  11. 2022CodeT: Code Generation with Generated Testscited by 369
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
38 more
  1. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  2. 2026Reinforcement Learning via Self-Distillationcited by 285
  3. 2022Generating Sequences by Learning to Self-Correctcited by 271
  4. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  5. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  6. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  7. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  8. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  9. 2023Planning with Large Language Models for Code Generationcited by 179
  10. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  11. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  12. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  13. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  14. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  15. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  16. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  17. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 130
  18. 2023Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generationcited by 128
  19. 2023RLTF: Reinforcement Learning from Unit Test Feedbackcited by 110
  20. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  21. 2024Large Language Models Meet NLP: A Surveycited by 102
  22. 2023Textually Pretrained Speech Language Modelscited by 99
  23. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 95
  24. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  25. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  26. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  27. 2024Can Language Models Solve Olympiad Programming?cited by 70
  28. 2023PanGu-Coder2: Boosting Large Language Models for Code with Ranking Feedbackcited by 69
  29. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
  30. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  31. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  32. 2023CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modulescited by 58
  33. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 56
  34. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  35. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  36. 2025FLEX: Continuous Agent Evolution via Forward Learning from Experiencecited by 51
  37. 2024Code Repair with LLMs gives an Exploration-Exploitation Tradeoffcited by 50
  38. 2023RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repaircited by 43

50 most cited of 342 All 342 (JSON)