# CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs

arXiv: https://arxiv.org/abs/2502.07980 · HTML: https://mlsanity.com/paper/2502.07980/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 11 Feb 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://mlsanity.com/paper/2406.06608/) (2024)
- [AnalogCoder: Analog Circuit Design via Training-Free Code Generation](https://mlsanity.com/paper/2405.14918/) (2024)
- [CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities](https://mlsanity.com/paper/2401.06961/) (2024)
- [Learning to Design Circuits](https://mlsanity.com/paper/1812.02734/) (2018)

## Built on by (most cited first, 11 of 11)

- [Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization](https://mlsanity.com/paper/2604.12290/) (2026)
- [AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuits](https://mlsanity.com/paper/2505.24138/) (2025)
- [EngTrace: A Symbolic Benchmark for Verifiable Process Supervision of Engineering Reasoning](https://mlsanity.com/paper/2511.01650/) (2025)
- [CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process](https://mlsanity.com/paper/2509.22339/) (2025)
- [Enhancing Large Language Models for Automated Homework Assessment in Undergraduate Circuit Analysis](https://mlsanity.com/paper/2511.18221/) (2025)
- [CircuChain: Disentangling Competence and Compliance in LLM Circuit Analysis](https://mlsanity.com/paper/2602.15037/) (2026)
- [NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation](https://mlsanity.com/paper/2608.12197/) (2026)
- [AMS-IO-Bench and AMS-IO-Agent: Benchmarking and Structured Reasoning for Analog and Mixed-Signal Integrated Circuit Input/Output Design](https://mlsanity.com/paper/2512.21613/) (2025)
- [VLM-CAD: VLM-Optimized Collaborative Agent Design Workflow for Analog Circuit Sizing](https://mlsanity.com/paper/2601.07315/) (2026)
- [EDU-CIRCUIT-HW: Evaluating Multimodal Large Language Models on Real-World University-Level STEM Student Handwritten Solutions](https://mlsanity.com/paper/2602.00095/) (2026)
- [Beyond Pixels: Vector-to-Graph Transformation for Reliable Schematic Auditing](https://mlsanity.com/paper/2602.11678/) (2026)

## Abstract (arXiv)

The role of Large Language Models (LLMs) has not been extensively explored in analog circuit design, which could benefit from a reasoning-based approach that transcends traditional optimization techniques. In particular, despite their growing relevance, there are no benchmarks to assess LLMs' reasoning capability about circuits. Therefore, we created the CIRCUIT dataset consisting of 510 question-answer pairs spanning various levels of analog-circuit-related subjects. The best-performing model on our dataset, GPT-4o, achieves 48.04% accuracy when evaluated on the final numerical answer. To evaluate the robustness of LLMs on our dataset, we introduced a unique feature that enables unit-test-like evaluation by grouping questions into unit tests. In this case, GPT-4o can only pass 27.45% of the unit tests, highlighting that the most advanced LLMs still struggle with understanding circuits, which requires multi-level reasoning, particularly when involving circuit topologies. This circuit-specific benchmark highlights LLMs' limitations, offering valuable insights for advancing their application in analog integrated circuit design.