# Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation

arXiv: https://arxiv.org/abs/2407.08473 · HTML: https://mlsanity.com/paper/2407.08473/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 11 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [ChatEDA: A Large Language Model Powered Autonomous Agent for EDA](https://mlsanity.com/paper/2308.10204/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [Prompting Is Programming: A Query Language for Large Language Models](https://mlsanity.com/paper/2212.06094/) (2022)
- [ChipGPT: How far are we from natural language hardware design](https://mlsanity.com/paper/2305.14019/) (2023)
- [RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models](https://mlsanity.com/paper/2311.16543/) (2023)
- [GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models](https://mlsanity.com/paper/2309.10730/) (2023)
- [Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework](https://mlsanity.com/paper/2403.11202/) (2024)
- [AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMs](https://mlsanity.com/paper/2402.00386/) (2024)
- [HDLdebugger: Streamlining HDL debugging with Large Language Models](https://mlsanity.com/paper/2403.11671/) (2024)
- [On the Viability of using LLMs for SW/HW Co-Design: An Example in Designing CiM DNN Accelerators](https://mlsanity.com/paper/2306.06923/) (2023)

## Built on by (most cited first, 11 of 11)

- [DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model](https://mlsanity.com/paper/2502.15832/) (2025)
- [AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMs](https://mlsanity.com/paper/2508.02518/) (2025)
- [RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs](https://mlsanity.com/paper/2507.16200/) (2025)
- [Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead](https://mlsanity.com/paper/2512.00020/) (2025)
- [Are LLMs Any Good for High-Level Synthesis?](https://mlsanity.com/paper/2408.10428/) (2024)
- [ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learning](https://mlsanity.com/paper/2507.04736/) (2025)
- [BRIDGES: Bridging Graph Modality and Large Language Models within EDA Tasks](https://mlsanity.com/paper/2504.05180/) (2025)
- [RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models](https://mlsanity.com/paper/2606.08976/) (2026)
- [ML For Hardware Design Interpretability: Challenges and Opportunities](https://mlsanity.com/paper/2504.08852/) (2025)
- [ATLAS: AI-Assisted Threat-to-Assertion Learning for System-on-Chip Security Verification](https://mlsanity.com/paper/2603.01170/) (2026)
- [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026)

## Abstract (arXiv)

Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singular and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large-hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.