# NotSoTiny: A Large, Living Benchmark for RTL Code Generation

arXiv: https://arxiv.org/abs/2512.20823 · HTML: https://mlsanity.com/paper/2512.20823/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 23 Dec 2025.

## Builds on (most cited first, 13 of 13)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models](https://mlsanity.com/paper/2402.15938/) (2024)
- [Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation](https://mlsanity.com/paper/2408.11053/) (2024)
- [A Survey of Research in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2501.09655/) (2025)
- [HaVen: Hallucination-Mitigated LLM for Verilog Code Generation Aligned with HDL Engineers](https://mlsanity.com/paper/2501.04908/) (2025)
- [Towards Data Contamination Detection for Modern Large Language Models: Limitations, Inconsistencies, and Oracle Challenges](https://mlsanity.com/paper/2409.09927/) (2024)
- [VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contamination](https://mlsanity.com/paper/2503.13572/) (2025)
- [TuRTLe: A Unified Evaluation of LLMs for RTL Generation](https://mlsanity.com/paper/2504.01986/) (2025)

## Built on by (most cited first, 4 of 4)

- [Exploring the Agentic Frontier of Verilog Code Generation](https://mlsanity.com/paper/2603.19347/) (2026)
- [RuC: HDL-Agnostic Rule Completion Benchmark Generation](https://mlsanity.com/paper/2604.27780/) (2026)
- [BEHAVE: Functional Behavior Modeling Enables Self-Improving Agents for Hardware Design and Verification](https://mlsanity.com/paper/2609.34785/) (2026)
- [SGAnalog: An End-to-End Circuit Benchmark from Open-Source Silicon Tapeouts](https://mlsanity.com/paper/2610.03934/) (2026)

## Abstract (arXiv)

LLMs have shown early promise in generating RTL code, yet evaluating their capabilities in realistic setups remains a challenge. So far, RTL benchmarks have been limited in scale, skewed toward trivial designs, offering minimal verification rigor, and remaining vulnerable to data contamination. To overcome these limitations and to push the field forward, this paper introduces NotSoTiny, a benchmark that assesses LLM on the generation of structurally rich and context-aware RTL. Built from hundreds of actual hardware designs produced by the Tiny Tapeout community, our automated pipeline removes duplicates, verifies correctness and periodically incorporates new designs to mitigate contamination, matching Tiny Tapeout release schedule. Evaluation results show that NotSoTiny tasks are more challenging than prior benchmarks, emphasizing its effectiveness in overcoming current limitations of LLMs applied to hardware design, and in guiding the improvement of such promising technology.