mlsanityData to 14 Oct 2026

Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback

arXiv 2504.15804 ↗First posted 22 Apr 2025cs.AR cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Large language models (LLMs) have shown strong performance in Verilog generation from natural language description. However, ensuring the functional correctness of the generated code remains a significant challenge. This paper introduces a method that integrates verification insights from testbench into the training of Verilog generation LLMs, aligning the training with the fundamental goal of hardware design: functional correctness. The main obstacle in using LLMs for Verilog code generation is the lack of sufficient functional verification data, particularly testbenches paired with design specifications and code. To address this problem, we introduce an automatic testbench generation pipeline that decomposes the process and uses feedback from the Verilog compiler simulator (VCS) to reduce hallucination and ensure correctness. We then use the testbench to evaluate the generated codes and collect them for further training, where verification insights are introduced. Our method applies reinforcement learning (RL), specifically direct preference optimization (DPO), to align Verilog code generation with functional correctness by training preference pairs based on testbench outcomes. In evaluations on VerilogEval-Machine, VerilogEval-Human, RTLLM v1.1, RTLLM v2, and VerilogEval v2, our approach consistently outperforms state-of-the-art baselines in generating functionally correct Verilog code. We open source all training code, data, and models at https://anonymous.4open.science/r/VeriPrefer-E88B.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 41 references that are in the map. First posted on arXiv 22 Apr 2025.

Builds on

32 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases11 Jun

Other references · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  12. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
19 more
  1. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 558
  2. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  3. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  4. 2023Tool Learning with Foundation Modelscited by 302
  5. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  6. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  7. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  8. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 101
  9. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 100
  10. 2023Execution-based Code Generation using Deep Reinforcement Learningcited by 99
  11. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Toolcited by 85
  12. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  13. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  14. 2024CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarizationcited by 38
  15. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMscited by 36
  16. 2025OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generationcited by 31
  17. 2025HaVen: Hallucination-Mitigated LLM for Verilog Code Generation Aligned with HDL Engineerscited by 25
  18. 2024PyraNet: A Multi-Layered Hierarchical Dataset for Verilogcited by 19
  19. 2024IRCoCo: Immediate Rewards-Guided Deep Reinforcement Learning for Code Completioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generationcited by 35
  2. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
  3. 2025iDSE: Navigating Design Space Exploration in High-Level Synthesis Using LLMscited by 7
  4. 2025QiMeng-SALV: Signal-Aware Learning for Verilog Code Generationcited by 4
  5. 2026SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generationcited by 4
  6. 2026CktEvo: Repository-Level RTL Code Benchmark for Design Evolutioncited by 3
  7. 2025VeriGRAG: Enhancing LLM-Based Verilog Code Generation with Structure-Aware Soft Promptscited by 2
  8. 2025QiMeng-CRUX: Narrowing the Gap Between Natural Language and Verilog via Core Refined Understanding eXpression for Circuit Designcited by 1
  9. 2026Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoningcited by 1
  10. 2026LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestrationcited by 1
  11. 2026The New Compiler Stack: A Survey on the Synergy of LLMs and Compilerscited by 0
  12. 2026Exploring LLM-based Verilog Code Generation with Data-Efficient Fine-Tuning and Testbench Automationcited by 0
2 more
  1. 2026CovR: Coverage-Aware Hardware Verification via Reasoning-Guided Reinforcement Learningcited by 0
  2. 2026BEHAVE: Functional Behavior Modeling Enables Self-Improving Agents for Hardware Design and Verificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation
  2. 2025VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning
  3. 2024Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning
  4. 2025TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generation
  5. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  6. 2026GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation
  7. 2025ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learning
  8. 2025VFocus: Better Verilog Generation from Large Language Model via Focused Reasoning