# Validation and Simulation Catch Different Errors: Four Levels of Evaluation for LLM-Generated Circuits

arXiv: https://arxiv.org/abs/2609.26830 · HTML: https://mlsanity.com/paper/2609.26830/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [AnalogCoder: Analog Circuit Design via Training-Free Code Generation](https://mlsanity.com/paper/2405.14918/) (2024)
- [AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologies](https://mlsanity.com/paper/2503.00205/) (2025)
- [Masala-CHAI: A Large-Scale SPICE Netlist Dataset for Analog Circuits by Harnessing AI](https://mlsanity.com/paper/2411.14299/) (2024)
- [AMSnet 2.0: A Large AMS Database with AI Segmentation for Net Detection](https://mlsanity.com/paper/2505.09155/) (2025)
- [LaMAGIC2: Advanced Circuit Formulations for Language Model-Based Analog Topology Generation](https://mlsanity.com/paper/2506.10235/) (2025)
- [AnalogAgent: Self-Improving Analog Circuit Design Automation with LLM Agents](https://mlsanity.com/paper/2603.23910/) (2026)
- [CktGen: Automated Analog Circuit Design with Generative Artificial Intelligence](https://mlsanity.com/paper/2410.00995/) (2024)
- [AnalogMaster: Large Language Model-based Automated Analog IC Design Framework from Image to Layout](https://mlsanity.com/paper/2604.20916/) (2026)

## Abstract (arXiv)

Simulation success is not equivalent to structural correctness for LLM-generated circuits. We define and measure four evaluation levels -- schema validity, topological validity, backend executability, and component-set agreement -- on a 150-circuit trilingual benchmark, through a deployed pipeline built on a typed circuit interchange representation. The levels are not nested. On gpt-4o-mini, 16 of 150 circuits (10.7%, 95% CI 6.7-16.6) were rejected by the topological validator but executed in ngspice with no error or warning; 12 of these contained exactly the requested components, with one terminal disconnected. Conversely, 7 circuits (4.7%) passed the validator and ngspice refused them. Ten failed both checks and 117 passed both, so each check detects a class the other misses. A minimal three-component divider shows the cost: a dangling resistor reports 5.00 V instead of 2.50 V while ngspice stays silent. A paired ablation, in which every arm is evaluated from the same model sample rather than a fresh one, separates each repair stage from sampling noise. On a stratified 45-circuit subsample, model repair raised topological validity from 40.0% to 84.4% (+20 circuits, no regressions) while moving executability by a net 6 (+7, -1), an effect this sample size does not resolve, and component agreement by 2. One circuit moved in opposite directions at two levels in a single repair step. Against a direct-netlist baseline the pipeline executed 88.7% against 47.3%, or 62.7% under an accounting that credits the baseline with every failure we cannot confidently attribute to the netlist. These results support a narrow methodological conclusion: structural validation and simulation should be reported as distinct evaluation stages for LLM-generated circuits. A circuit that runs is not necessarily structurally valid, and a structurally valid circuit is not necessarily executable.