mlsanityData to 7 Oct 2026

CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

arXiv 2409.12993 ↗First posted 19 Sep 2024cs.AR cs.CLcited by 41

On the map: Synthetic data and self-improvement for LLMs → Code and instruction data synthesis

Abstract (arXiv)

Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog coding, with synthetic data from prior methods. We identify two main issues: difficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making "minor" mistakes. To address these limitations, we enhance data curation by creating correct-by-construction data targeting non-textual representations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open-source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-art results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Code and instruction data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 52 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

37 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 36

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
24 more
  1. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  2. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  3. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  4. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  5. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  6. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  7. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  8. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  9. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  10. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  11. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  12. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  13. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 143
  14. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  15. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 99
  16. 2023RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Modelscited by 98
  17. 2024OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflectioncited by 56
  18. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  19. 2024Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTScited by 42
  20. 2024AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Designcited by 38
  21. 2024CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarizationcited by 38
  22. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
  23. 2024LLM-Aided Testbench Generation and Bug Detection for Finite-State Machinescited by 15
  24. 2024AutoCoder: Enhancing Code Large Language Model with textscAIEV-Instructcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 26; 2026: 15.

41 papers.

Sort:

By area:

  1. 2025Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verificationcited by 41
  2. 2025QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generationcited by 34
  3. 2025ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generationcited by 22
  4. 2025RTL++: Graph-enhanced LLM for RTL Code Generationcited by 16
  5. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  6. 2025AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMscited by 14
  7. 2026ACE-RTL: When Agentic Context Evolution Meets RTL-Specialized LLMscited by 12
  8. 2025VeriCoder: Enhancing LLM-Based RTL Code Generation through Functional Correctness Validationcited by 11
  9. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 10
  10. 2025VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generationcited by 8
  11. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 7
  12. 2026Agentic Hardware Design as Repository-Level Code Evolutioncited by 6
29 more
  1. 2025ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learningcited by 5
  2. 2025Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selectioncited by 4
  3. 2025DecoRTL: A Run-time Decoding Framework for RTL Code Generation with LLMscited by 4
  4. 2025REvolution: An Evolutionary Framework for RTL Generation driven by Large Language Modelscited by 4
  5. 2025Free and Fair Hardware: A Pathway to Copyright Infringement-Free Verilog Generation using LLMscited by 2
  6. 2025hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generationcited by 2
  7. 2026OSIRIS: Bridging Analog Circuit Design and Machine Learning with Scalable Dataset Generationcited by 2
  8. 2025Beyond Tokens: Enhancing RTL Quality Estimation via Structural Graph Learningcited by 1
  9. 2025DeepV: A Model-Agnostic Retrieval-Augmented Framework for Verilog Code Generation with a High-Quality Knowledge Basecited by 1
  10. 2025Faver: Boosting LLM-based RTL Generation with Function Abstracted Verifiable Middlewarecited by 1
  11. 2025Multimodal Chip Physical Design Engineer Assistantcited by 1
  12. 2025QiMeng-CRUX: Narrowing the Gap Between Natural Language and Verilog via Core Refined Understanding eXpression for Circuit Designcited by 1
  13. 2026QiMeng-ChipV-RTL: Exploiting Information Locality for IP-level Verilog Generationcited by 1
  14. 2026TimingLLM: A Two-Stage Retrieval-Augmented Framework for Pre-Synthesis Timing Prediction from Verilogcited by 1
  15. 2025RocketPPA: Code-Level Power, Performance, and Area Prediction via LLM and Mixture of Expertscited by 0
  16. 2025QiMeng: Fully Automated Hardware and Software Design for Processor Chipcited by 0
  17. 2025Guaranteed Guess: A Language Modeling Approach for CISC-to-RISC Transpilation with Testing Guaranteescited by 0
  18. 2025VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learningcited by 0
  19. 2025When Forgetting Builds Reliability: LLM Unlearning for Reliable Hardware Code Generationcited by 0
  20. 2026ChipCraftBrain: Validation-First RTL Generation via Multi-Agent Orchestrationcited by 0
  21. 2026UniRTL: Unifying Code and Graph for Robust RTL Representation Learningcited by 0
  22. 2026CASS-RTL: Correctness-Aware Subspace Steering for RTL Generation with LLMscited by 0
  23. 2026EstRTL: Functional Estimation Guided RTL Code Generationcited by 0
  24. 2026LLM4RTL: Tool-Assisted LLM for RTL Generationcited by 0
  25. 2026How LLMs Fail and Generalize in RTL Coding for Hardware Design?cited by 0
  26. 2026VeriPilot: An LLM-Powered Verilog Debugging Frameworkcited by 0
  27. 2026SCALE: Self-Supervised Constraint-Aware Layout GEneration for Local P&R DRV Fixing at Advanced Nodescited by 0
  28. 2026VeriTrace: Human-Like Temporal Exploration Completes Agentic Action Spacecited by 0
  29. 2026LLMs in Digital EDA: A perspective on shifting roles from Generation to Orchestrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMs
  2. 2025Speculative Decoding for Verilog: Speed and Quality, All in One
  3. 2025VFocus: Better Verilog Generation from Large Language Model via Focused Reasoning
  4. 2025VeriMind: Agentic LLM for Automated Verilog Generation with a Novel Evaluation Metric
  5. 2025Insights from Verification: Training a Verilog Generation LLM with Reinforcement Learning with Testbench Feedback
  6. 2026OpenRTLSet: A Fully Open-Source Dataset for Large Language Model-based Verilog Module Design
  7. 2026SiliconMind-V1: Multi-Agent Distillation and Debug-Reasoning Workflows for Verilog Code Generation
  8. 2026Execution-Anchored Hallucination Calibration Reranking for Verilog Code Generation ↗