# Zero-Shot RTL Code Generation with Attention Sink Augmented Large Language Models

arXiv: https://arxiv.org/abs/2401.08683 · HTML: https://mlsanity.com/paper/2401.08683/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [VeriGen: A Large Language Model for Verilog Code Generation](https://mlsanity.com/paper/2308.00708/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [Benchmarking Large Language Models for Automated Verilog RTL Code Generation](https://mlsanity.com/paper/2212.11140/) (2022)
- [ChipGPT: How far are we from natural language hardware design](https://mlsanity.com/paper/2305.14019/) (2023)
- [GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models](https://mlsanity.com/paper/2309.10730/) (2023)
- [DAVE: Deriving Automatically Verilog from English](https://mlsanity.com/paper/2009.01026/) (2020)

## Built on by (most cited first, 5 of 5)

- [Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation](https://mlsanity.com/paper/2604.10098/) (2026)
- [SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models](https://mlsanity.com/paper/2406.05678/) (2024)
- [Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sink](https://mlsanity.com/paper/2602.05228/) (2026)
- [ML For Hardware Design Interpretability: Challenges and Opportunities](https://mlsanity.com/paper/2504.08852/) (2025)
- [PEFA-AI: Advancing Open-source LLMs for RTL generation using Progressive Error Feedback Agentic-AI](https://mlsanity.com/paper/2511.03934/) (2025)

## Abstract (arXiv)

The design and optimization of hardware have traditionally been resource-intensive, demanding considerable expertise and dependence on established design automation tools. This paper discusses the possibility of exploiting large language models to streamline the code generation process in hardware design. In contrast to earlier studies, this paper aims to use large language models that accepts high-level design specifications through a single prompt to generate corresponding Register-Transfer Level (RTL) code. The ability to use large language models on RTL code generation not only expedites design iteration cycles but also facilitates the exploration of design spaces that have computational challenges for conventional techniques. Through our evaluation, we demonstrate the shortcoming of existing attention mechanisms, and present the abilities of language models to produce functional, optimized, and industry-standard compliant RTL code when a novel attention mechanism is used. These findings underscore the expanding role of large language models in shaping the future landscape of architectural exploration and automation in hardware design.