mlsanityData to 7 Oct 2026

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

arXiv 2412.14308 ↗First posted 18 Dec 2024cs.SE cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Software testing is a crucial but time-consuming aspect of software development, and recently, Large Language Models (LLMs) have gained popularity for automated test case generation. However, because LLMs are trained on vast amounts of open-source code, they often generate test cases that do not adhere to best practices and may even contain test smells (anti-patterns). To address this issue, we propose Reinforcement Learning from Static Quality Metrics (RLSQM), wherein we utilize Reinforcement Learning to generate high-quality unit tests based on static analysis-based quality metrics. First, we analyzed LLM-generated tests and show that LLMs frequently do generate undesirable test smells -- up to 37% of the time. Then, we implemented lightweight static analysis-based reward model and trained LLMs using this reward model to optimize for five code quality metrics. Our experimental results demonstrate that the RL-optimized Codex model consistently generated higher-quality test cases than the base LLM, improving quality metrics by up to 23%, and generated nearly 100% syntactically-correct code. RLSQM also outperformed GPT-4 on all code quality metrics, in spite of training a substantially cheaper Codex model. We provide insights into how reliably utilize RL to improve test generation quality and show that RLSQM is a significant step towards enhancing the overall efficiency and reliability of automated software testing. Our data are available at https://doi.org/10.6084/m9.figshare.25983166.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 68 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

35 of 68 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
23 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  9. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  12. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  13. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  14. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 193
  15. 2020Unit Test Case Generation with Transformers and Focal Contextcited by 118
  16. 2023No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generationcited by 82
  17. 2021TOGA: A Neural Method for Test Oracle Generationcited by 57
  18. 2023Effective Test Generation Using Pre-trained Large Language Models and Mutation Testingcited by 51
  19. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  20. 2023A3Test: Assertion-Augmented Automated Test Case Generationcited by 33
  21. 2023Can Large Language Models Write Good Property-Based Tests?cited by 30
  22. 2021Learning to Complete Code with Sketchescited by 23
  23. 2023Transformer-based Vulnerability Detection in Code at EditTime: Zero-shot, Few-shot, or Fine-tuning?cited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
  2. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  3. 2024Process Supervision-Guided Policy Optimization for Code Generation
  4. 2025CodeContests+: High-Quality Test Case Generation for Competitive Programming
  5. 2026SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
  6. 2025HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
  7. 2025Integrating Symbolic Execution into the Fine-Tuning of Code-Generating LLMs
  8. 2025HardTests: Synthesizing High-Quality Test Cases for LLM Coding