mlsanityData to 14 Oct 2026

ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design

arXiv 2601.21448 ↗First posted 29 Jan 2026cs.AI cs.ARcited by 8

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) show significant potential in hardware engineering, current benchmarks suffer from saturation and limited task diversity, failing to reflect LLMs' performance in real industrial workflows. To address this gap, we propose a comprehensive benchmark for AI-aided chip design that rigorously evaluates LLMs across three critical tasks: Verilog generation, debugging, and reference model generation. Our benchmark features 44 realistic modules with complex hierarchical structures, 89 systematic debugging cases, and 132 reference model samples across Python, SystemC, and CXXRTL. Evaluation results reveal substantial performance gaps, with state-of-the-art Claude-4.5-opus achieving only 30.74% on Verilog generation and 13.33% on Python reference model generation, demonstrating significant challenges compared to existing saturated benchmarks where SOTA models achieve over 95% pass rates. Additionally, to help enhance LLM reference model generation, we provide an automated toolbox for high-quality training data generation, facilitating future research in this underexplored domain. Our code is available at https://github.com/zhongkaiyu/ChipBench.git.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 27 of its 53 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

27 of 53 references.

References · 27

  1. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  2. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  3. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  4. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  5. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
  6. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  7. 2025A Survey on Code Generation with LLM-based Agentscited by 106
  8. 2023The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Developmentcited by 96
  9. 2024VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Toolcited by 85
  10. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimizationcited by 84
  11. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  12. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
15 more
  1. 2023On the Effectiveness of Large Language Models in Domain-Specific Code Generationcited by 44
  2. 2023Large Language Model-Aware In-Context Learning for Code Generationcited by 43
  3. 2024AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Designcited by 41
  4. 2025QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generationcited by 35
  5. 2024MAGE: A Multi-Agent Engine for Automated RTL Code Generationcited by 31
  6. 2025OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generationcited by 31
  7. 2024CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Designcited by 24
  8. 2024PyraNet: A Multi-Layered Hierarchical Dataset for Verilogcited by 19
  9. 2025Spec2RTL-Agent: Automated Hardware Code Generation from Complex Specifications Using LLM Agent Systemscited by 18
  10. 2024UVLLM: An Automated Universal RTL Verification Framework using LLMscited by 17
  11. 2024A Multi-Expert Large Language Model Architecture for Verilog Code Generationcited by 15
  12. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  13. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  14. 2025Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysiscited by 10
  15. 2024Evaluating Large Language Models for Automatic Register Transfer Logic Generation via High-Level Synthesiscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026ChipMATE: Multi-Agent Training via Reinforcement Learning for Enhanced RTL Generationcited by 1
  2. 2026ChipCraftBrain: Validation-First RTL Generation via Multi-Agent Orchestrationcited by 0
  3. 2026Think Through a Bottleneck: Hourglass Reasoning for Rigorous Inductioncited by 0
  4. 2026CLOSER-Bench: Evaluating Budgeted Cross-Stage Design Closure for Hardware Agentscited by 0
  5. 2026WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveformscited by 0
  6. 2026PerfReasoning: How Well Do LLMs Reason on Hardware Performance?cited by 0
  7. 2026BEHAVE: Functional Behavior Modeling Enables Self-Improving Agents for Hardware Design and Verificationcited by 0
  8. 2026Large-scale factor analysis shows machine intelligence is only partially interpretablecited by 0