mlsanityData to 14 Oct 2026

VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination

arXiv 2503.13572 ↗First posted 17 Mar 2025cs.AR cs.CR cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized code generation, achieving exceptional results on various established benchmarking frameworks. However, concerns about data contamination - where benchmark data inadvertently leaks into pre-training or fine-tuning datasets - raise questions about the validity of these evaluations. While this issue is known, limiting the industrial adoption of LLM-driven software engineering, hardware coding has received little to no attention regarding these risks. For the first time, we analyze state-of-the-art (SOTA) evaluation frameworks for Verilog code generation (VerilogEval and RTLLM), using established methods for contamination detection (CCD and Min-K% Prob). We cover SOTA commercial and open-source LLMs (CodeGen2.5, Minitron 4b, Mistral 7b, phi-4 mini, LLaMA-{1,2,3.1}, GPT-{2,3.5,4o}, Deepseek-Coder, and CodeQwen 1.5), in baseline and fine-tuned models (RTLCoder and Verigen). Our study confirms that data contamination is a critical concern. We explore mitigations and the resulting trade-offs for code quality vs fairness (i.e., reducing contamination toward unbiased benchmarking).

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 44 of its 57 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

44 of 57 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Mistral 7Bcited by 3,164
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  9. 2025Large Language Diffusion Modelscited by 912
  10. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  11. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 387
32 more
  1. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 321
  2. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  3. 2022Data Contamination: From Memorization to Exploitationcited by 236
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  5. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  6. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  7. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  8. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  10. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  11. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  12. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  13. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 122
  14. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  15. 2023ChipGPT: How far are we from natural language hardware designcited by 105
  16. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 100
  17. 2023AutoChip: Automating HDL Generation Using LLM Feedbackcited by 83
  18. 2023RTLCoder: Outperforming GPT-3.5 in Design RTL Generation with Our Open-Source Dataset and Lightweight Solutioncited by 71
  19. 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
  20. 2023Can we trust the evaluation on ChatGPT?cited by 63
  21. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 61
  22. 2023An Open Source Data Contamination Report for Large Language Modelscited by 51
  23. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 45
  24. 2024AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Designcited by 41
  25. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challengescited by 32
  26. 2024AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMscited by 31
  27. 2024TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMscited by 24
  28. 2024Investigating the Impact of Data Contamination of Large Language Models in Text-to-SQL Translationcited by 21
  29. 2024LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trustcited by 20
  30. 2024LLM-Aided Testbench Generation and Bug Detection for Finite-State Machinescited by 17
  31. 2024RTL-Breaker: Assessing the Security of LLMs against Backdoor Attacks on HDL Code Generationcited by 15
  32. 2024Discovering and exploring cases of educational source code plagiarism with Doloscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025SALAD: Systematic Assessment of Machine Unlearning on LLM-Aided Hardware Designcited by 12
  2. 2025NetDeTox: Adversarial and Efficient Evasion of Hardware-Security GNNs via RL-LLM Orchestrationcited by 9
  3. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
  4. 2025VeriDispatcher: Multi-Model Dispatching through Pre-Inference Difficulty Prediction for RTL Generation Optimizationcited by 5
  5. 2025NotSoTiny: A Large, Living Benchmark for RTL Code Generationcited by 4
  6. 2026GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generationcited by 4
  7. 2026TrojanGYM: A Detector-in-the-Loop LLM for Adaptive RTL Hardware Trojan Insertioncited by 2
  8. 2025Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalizationcited by 1
  9. 2026GUIDE: GenAI Units In Digital Design Educationcited by 1
  10. 2026LLMs for Secure Hardware Design and Related Problems: Opportunities and Challengescited by 1
  11. 2026Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Modelscited by 0
  12. 2026Hardware Design and Security in the Era of Chiplets and LLMscited by 0