# NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation

arXiv: https://arxiv.org/abs/2608.12197 · HTML: https://mlsanity.com/paper/2608.12197/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [AnalogCoder: Analog Circuit Design via Training-Free Code Generation](https://mlsanity.com/paper/2405.14918/) (2024)
- [The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs](https://mlsanity.com/paper/2509.09677/) (2025)
- [AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologies](https://mlsanity.com/paper/2503.00205/) (2025)
- [Masala-CHAI: A Large-Scale SPICE Netlist Dataset for Analog Circuits by Harnessing AI](https://mlsanity.com/paper/2411.14299/) (2024)
- [AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuits](https://mlsanity.com/paper/2505.24138/) (2025)
- [Schemato -- An LLM for Netlist-to-Schematic Conversion](https://mlsanity.com/paper/2411.13899/) (2024)
- [CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs](https://mlsanity.com/paper/2502.07980/) (2025)
- [SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidance](https://mlsanity.com/paper/2410.20553/) (2024)
- [ALIGN: A System for Automating Analog Layout](https://mlsanity.com/paper/2008.10682/) (2020)
- [SPICED: Syntactical Bug and Trojan Pattern Identification in A/MS Circuits using LLM-Enhanced Detection](https://mlsanity.com/paper/2408.16018/) (2024)
- [Image2Net: Datasets, Benchmark and Hybrid Framework to Convert Analog Circuit Diagrams into Netlists](https://mlsanity.com/paper/2508.13157/) (2025)
- [TopoSizing: An LLM-aided Framework of Topology-based Understanding and Sizing for AMS Circuits](https://mlsanity.com/paper/2509.14169/) (2025)
- [Evaluating LLM-based Workflows for Switched-Mode Power Supply Design](https://mlsanity.com/paper/2507.10639/) (2025)
- [CircuitFormer: A Circuit Language Model for Analog Topology Design from Natural Language Prompt](https://mlsanity.com/paper/2605.05773/) (2026)

## Built on by (most cited first, 1 of 1)

- [Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks](https://mlsanity.com/paper/2609.33356/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used in circuit design workflows, yet their reliability on simulator-facing SPICE netlist recognition and manipulation remains poorly understood and is rarely separated from high-level design reasoning. Although netlists are textual, they encode structured circuit objects through topology and parameters. We present NetlistBench, a structure-verified benchmark for SPICE netlist recognition and manipulation. NetlistBench contains 2,342 cases across 24 task families, covering parameter and connectivity recognition and edits, hierarchical operations, equivalence judgment, and long-horizon compound editing. Model outputs are evaluated by a deterministic structure-aware oracle. Across six non-thinking LLMs, performance varies substantially with operation-level structural complexity. Simple local edits reach 96%--100% accuracy, while device addition drops to 41%--83% and equivalence judgment to 49%--90%. Enabling reasoning substantially improves weaker models but does not eliminate structure-preservation failures, with performance still degrading sharply as the edit horizon increases. NetlistBench identifies netlist reliability as a distinct bottleneck for trustworthy LLM-based circuit design automation.