Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks
New in LLM agents and tool use; no direction yet. LLM agents and tool use
Abstract (arXiv)
Coding agents now sustain hours-long, tool-driven loops, yet their ability to carry long-horizon analog and mixed-signal circuits to electrical specification remains unmeasured. We introduce Analog Design Bench, a long-horizon agentic benchmark of 50 transistor-level design tasks contributed by 17 chip designers. Agents work with an open-source simulator, while an isolated verifier evaluates the submitted circuit using specification-based electrical tests. We evaluate 15 agent configurations across 2,250 two-hour attempts and observe full-specification pass rates from 8.0% to 78.0%. Coding-benchmark performance correlates with analog results but leaves much of the performance spread unexplained. Our failure analysis shows that most unsuccessful submissions have no recorded legality rejection but fail electrical acceptance, identifying electrical closure as the dominant endpoint challenge. We test time, reasoning effort, agent harness, and supplied design knowledge as interventions. Longer budgets and higher reasoning effort improve performance, while general skill documents provide little benefit and sometimes reduce performance. Supplying a task-matched reference topology, an idealized form of circuit-IP retrieval, raises DeepSeek V4 Pro by 18.7 percentage points and mainly accelerates GPT-5.6 Sol.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 70 references that are in the map. First posted on arXiv 27 Sep 2026.
Builds on
40 of 70 references.
References · 40
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
- 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
- 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
- 2020GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learningcited by 89
- 2024AnalogCoder: Analog Circuit Design via Training-Free Code Generationcited by 84
- 2023CktGNN: Circuit Graph Neural Network for Electronic Design Automationcited by 58
- 2020AutoCkt: Deep Reinforcement Learning of Analog Circuit Designscited by 51
28 more
- 2026DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Taskscited by 49
- 2024ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Modelscited by 44
- 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 41
- 2025AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologiescited by 37
- 2024LaMAGIC: Language-Model-based Topology Generation for Analog Integrated Circuitscited by 33
- 2024AmpAgent: An LLM-based Multi-Agent System for Multi-stage Amplifier Schematic Design from Literature for Process and Performance Portingcited by 29
- 2024AMSNet: Netlist Dataset for AMS Circuitscited by 25
- 2024LEDRO: LLM-Enhanced Design Space Reduction and Optimization for Analog Circuitscited by 22
- 2024AnalogXpert: Automating Analog Topology Synthesis by Incorporating Circuit Design Expertise into Large Language Modelscited by 18
- 2024AMSnet-KG: A Netlist Dataset for LLM-based AMS Circuit Auto-Design Using Knowledge Graph RAGcited by 16
- 2025AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuitscited by 15
- 2025AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMscited by 15
- 2025AnaFlow: Agentic LLM-based Workflow for Reasoning-Driven Explainable and Sample-Efficient Analog Circuit Sizingcited by 11
- 2024SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidancecited by 10
- 2026Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Surveycited by 10
- 2024AICircuit: A Multi-Level Dataset and Benchmark for AI-Driven Analog Integrated Circuit Designcited by 9
- 2024AnalogGym: An Open and Practical Testing Suite for Analog Circuit Synthesiscited by 9
- 2025TopoSizing: An LLM-aided Framework of Topology-based Understanding and Sizing for AMS Circuitscited by 7
- 2025The Dawn of Agentic EDA: A Survey of Autonomous Digital Chip Designcited by 7
- 2025LaMAGIC2: Advanced Circuit Formulations for Language Model-Based Analog Topology Generationcited by 6
- 2025HeaRT: A Hierarchical Circuit Reasoning Tree-Based Agentic Framework for AMS Design Optimizationcited by 5
- 2025AnalogTester: A Large Language Model-Based Framework for Automatic Testbench Generation in Analog Circuit Designcited by 4
- 2026AnalogAgent: Self-Improving Analog Circuit Design Automation with LLM Agentscited by 4
- 2025CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Processcited by 3
- 2025White-Box Reasoning: Synergizing LLM Strategy and gm/Id Data for Automated Analog Circuit Designcited by 1
- 2026AnalogRetriever: Learning Cross-Modal Representations for Analog Circuit Retrievalcited by 1
- 2026RF-Agent: A Practical Framework for Building Language Agents for RFIC Designcited by 1
- 2026NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025AnalogSAGE: Self-evolving Analog Design Multi-Agents with Stratified Memory and Grounded Experience
- 2026AgenticSizing: A Large Language Model-based Multi-Agent Framework for Analog Circuit Sizing
- 2026RFChipAgent: Multi-Agentic AI Flow for Analog/RF Chip Design
- 2026Analog-DB: An Agent-First Analog Integrated Circuit Database, From Blocks to Systems
- 2026AaLLM: An End-to-End Analog Circuit Design Framework from Topology Generation to Sizing Using Large Language Models
- 2026Can an Actor-Critic Optimization Framework Improve Analog Design?
- 2025AMS-IO-Bench and AMS-IO-Agent: Benchmarking and Structured Reasoning for Analog and Mixed-Signal Integrated Circuit Input/Output Design
- 2025CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs