# EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generation

arXiv: https://arxiv.org/abs/2511.12033 · HTML: https://mlsanity.com/paper/2511.12033/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 29 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Builds on (most cited first, 27 of 27)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [ChipGPT: How far are we from natural language hardware design](https://mlsanity.com/paper/2305.14019/) (2023)
- [RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models](https://mlsanity.com/paper/2311.16543/) (2023)
- [BetterV: Controlled Verilog Generation with Discriminative Guidance](https://mlsanity.com/paper/2402.03375/) (2024)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflection](https://mlsanity.com/paper/2407.16237/) (2024)
- [Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation](https://mlsanity.com/paper/2408.11053/) (2024)
- [A Survey of Research in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2501.09655/) (2025)
- [Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR](https://mlsanity.com/paper/2507.15778/) (2025)
- [AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMs](https://mlsanity.com/paper/2407.18333/) (2024)
- [AssertLLM: Generating Hardware Verification Assertions from Design Specifications via Multi-LLMs](https://mlsanity.com/paper/2411.14436/) (2024)
- [HaVen: Hallucination-Mitigated LLM for Verilog Code Generation Aligned with HDL Engineers](https://mlsanity.com/paper/2501.04908/) (2025)
- [Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning](https://mlsanity.com/paper/2508.02260/) (2025)
- [VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generation](https://mlsanity.com/paper/2505.11849/) (2025)
- [Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning](https://mlsanity.com/paper/2407.18271/) (2024)
- [PLUM: Improving Code LMs with Execution-Guided On-Policy Preference Learning Driven By Synthetic Test Cases](https://mlsanity.com/paper/2406.06887/) (2024)

## Built on by (most cited first, 2 of 2)

- [ChipMEM: Verification-Grounded Memory for EDA Agents](https://mlsanity.com/paper/2609.27067/) (2026)
- [BEHAVE: Functional Behavior Modeling Enables Self-Improving Agents for Hardware Design and Verification](https://mlsanity.com/paper/2609.34785/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) have demonstrated significant potential in hardware design automation, particularly in using natural language to synthesize Register-Transfer Level (RTL) code. Despite this progress, a gap remains between model capability and the demands of real-world RTL design, including syntax errors, functional hallucinations, and weak alignment to designer intent. Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising approach to bridge this gap, as hardware provides executable and formally checkable signals that can be used to further align model outputs with design intent. However, in long, structured RTL code sequences, not all tokens contribute equally to functional correctness, and na"ively spreading gradients across all tokens dilutes learning signals. A key insight from our entropy analysis in RTL generation is that only a small fraction of tokens (e.g., always, if, assign, posedge) exhibit high uncertainty and largely influence control flow and module structure. To address these challenges, we present EARL, an Entropy-Aware Reinforcement Learning framework for Verilog generation. EARL performs policy optimization using verifiable reward signals and introduces entropy-guided selective updates that gate policy gradients to high-entropy tokens. This approach preserves training stability and concentrates gradient updates on functionally important regions of code. Our experiments on VerilogEval and RTLLM show that EARL improves functional pass rates over prior LLM baselines by up to 14.7%, while reducing unnecessary updates and improving training stability. These results indicate that focusing RL on critical, high-uncertainty tokens enables more reliable and targeted policy improvement for structured RTL code generation.