# CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

arXiv: https://arxiv.org/abs/2409.12993 · HTML: https://mlsanity.com/paper/2409.12993/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 37 of its 52 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/)

## Builds on (most cited first, 30 of 37)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [The Stack: 3 TB of permissively licensed source code](https://mlsanity.com/paper/2211.15533/) (2022)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models](https://mlsanity.com/paper/2312.06585/) (2023)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training](https://mlsanity.com/paper/2309.17179/) (2023)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://mlsanity.com/paper/2208.08227/) (2022)
- [RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models](https://mlsanity.com/paper/2311.16543/) (2023)
- [BetterV: Controlled Verilog Generation with Discriminative Guidance](https://mlsanity.com/paper/2402.03375/) (2024)
- [OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflection](https://mlsanity.com/paper/2407.16237/) (2024)

## Built on by (most cited first, 30 of 42)

- [Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification](https://mlsanity.com/paper/2506.14074/) (2025)
- [QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation](https://mlsanity.com/paper/2505.24183/) (2025)
- [ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation](https://mlsanity.com/paper/2506.05566/) (2025)
- [RTL++: Graph-enhanced LLM for RTL Code Generation](https://mlsanity.com/paper/2505.13479/) (2025)
- [AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMs](https://mlsanity.com/paper/2508.02518/) (2025)
- [RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs](https://mlsanity.com/paper/2507.16200/) (2025)
- [VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validation](https://mlsanity.com/paper/2504.15659/) (2025)
- [SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Design](https://mlsanity.com/paper/2506.02089/) (2025)
- [ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMs](https://mlsanity.com/paper/2602.10218/) (2026)
- [VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generation](https://mlsanity.com/paper/2507.13369/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [Agentic Hardware Design as Repository-Level Code Evolution](https://mlsanity.com/paper/2606.28279/) (2026)
- [DecoRTL: A Run-time Decoding Framework for RTL Code Generation with LLMs](https://mlsanity.com/paper/2507.02226/) (2025)
- [ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learning](https://mlsanity.com/paper/2507.04736/) (2025)
- [Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selection](https://mlsanity.com/paper/2505.07309/) (2025)
- [REvolution: An Evolutionary Framework for RTL Generation driven by Large Language Models](https://mlsanity.com/paper/2510.21407/) (2025)
- [OSIRIS: Bridging Analog Circuit Design and Machine Learning with Scalable Dataset Generation](https://mlsanity.com/paper/2601.19439/) (2026)
- [Free and Fair Hardware: A Pathway to Copyright Infringement-Free Verilog Generation using LLMs](https://mlsanity.com/paper/2505.06096/) (2025)
- [hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation](https://mlsanity.com/paper/2506.04544/) (2025)
- [Beyond Tokens: Enhancing RTL Quality Estimation via Structural Graph Learning](https://mlsanity.com/paper/2508.18730/) (2025)
- [DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Base](https://mlsanity.com/paper/2510.05327/) (2025)
- [Faver: Boosting LLM-based RTL Generation with Function Abstracted Verifiable Middleware](https://mlsanity.com/paper/2510.08664/) (2025)
- [Multimodal Chip Physical Design Engineer Assistant](https://mlsanity.com/paper/2510.15872/) (2025)
- [QiMeng-CRUX: Narrowing the Gap Between Natural Language and Verilog via Core Refined Understanding eXpression for Circuit Design](https://mlsanity.com/paper/2511.20099/) (2025)
- [When Forgetting Builds Reliability: LLM Unlearning for Reliable Hardware Code Generation](https://mlsanity.com/paper/2512.05341/) (2025)
- [QiMeng-ChipV-RTL: Exploiting Information Locality for IP-level Verilog Generation](https://mlsanity.com/paper/2602.00704/) (2026)
- [TimingLLM: A Two-Stage Retrieval-Augmented Framework for Pre-Synthesis Timing Prediction from Verilog](https://mlsanity.com/paper/2604.23602/) (2026)
- [VeriPilot: An LLM-Powered Verilog Debugging Framework](https://mlsanity.com/paper/2606.23759/) (2026)
- [LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestration](https://mlsanity.com/paper/2608.27184/) (2026)
- [RocketPPA: Code-Level Power, Performance, and Area Prediction via LLM and Mixture of Experts](https://mlsanity.com/paper/2503.21971/) (2025)

## Abstract (arXiv)

Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog coding, with synthetic data from prior methods. We identify two main issues: difficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making "minor" mistakes. To address these limitations, we enhance data curation by creating correct-by-construction data targeting non-textual representations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open-source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-art results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM.