mlsanityData to 14 Oct 2026

Validation and Simulation Catch Different Errors: Four Levels of Evaluation for LLM-Generated Circuits

arXiv 2609.26830 ↗First posted 21 Sep 2026cs.AR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Simulation success is not equivalent to structural correctness for LLM-generated circuits. We define and measure four evaluation levels -- schema validity, topological validity, backend executability, and component-set agreement -- on a 150-circuit trilingual benchmark, through a deployed pipeline built on a typed circuit interchange representation. The levels are not nested. On gpt-4o-mini, 16 of 150 circuits (10.7%, 95% CI 6.7-16.6) were rejected by the topological validator but executed in ngspice with no error or warning; 12 of these contained exactly the requested components, with one terminal disconnected. Conversely, 7 circuits (4.7%) passed the validator and ngspice refused them. Ten failed both checks and 117 passed both, so each check detects a class the other misses. A minimal three-component divider shows the cost: a dangling resistor reports 5.00 V instead of 2.50 V while ngspice stays silent. A paired ablation, in which every arm is evaluated from the same model sample rather than a fresh one, separates each repair stage from sampling noise. On a stratified 45-circuit subsample, model repair raised topological validity from 40.0% to 84.4% (+20 circuits, no regressions) while moving executability by a net 6 (+7, -1), an effect this sample size does not resolve, and component agreement by 2. One circuit moved in opposite directions at two levels in a single repair step. Against a direct-netlist baseline the pipeline executed 88.7% against 47.3%, or 62.7% under an accounting that credits the baseline with every failure we cannot confidently attribute to the netlist. These results support a narrow methodological conclusion: structural validation and simulation should be reported as distinct evaluation stages for LLM-generated circuits. A circuit that runs is not necessarily structurally valid, and a structurally valid circuit is not necessarily executable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

15 of 18 references.

References · 15

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2023Teaching Large Language Models to Self-Debugcited by 988
  4. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  5. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  6. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  7. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  8. 2024AnalogCoder: Analog Circuit Design via Training-Free Code Generationcited by 84
  9. 2025AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologiescited by 37
  10. 2024Masala-CHAI: A Large-Scale SPICE Netlist Dataset for Analog Circuits by Harnessing AIcited by 18
  11. 2025AMSnet 2.0: A Large AMS Database with AI Segmentation for Net Detectioncited by 9
  12. 2025LaMAGIC2: Advanced Circuit Formulations for Language Model-Based Analog Topology Generationcited by 6
3 more
  1. 2026AnalogAgent: Self-Improving Analog Circuit Design Automation with LLM Agentscited by 4
  2. 2024CktGen: Automated Analog Circuit Design with Generative Artificial Intelligencecited by 2
  3. 2026AnalogMaster: Large Language Model-based Automated Analog IC Design Framework from Image to Layoutcited by 1

Built on by

Not cited yet.