# AMSbench: A Comprehensive Benchmark for Evaluating MLLM Capabilities in AMS Circuits

arXiv: https://arxiv.org/abs/2505.24138 · HTML: https://mlsanity.com/paper/2505.24138/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 30 of 30)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement](https://mlsanity.com/paper/2409.12122/) (2024)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [A Survey on Multimodal Large Language Models for Autonomous Driving](https://mlsanity.com/paper/2311.12320/) (2023)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [GroundingGPT:Language Enhanced Multi-modal Grounding Model](https://mlsanity.com/paper/2401.06071/) (2024)
- [AnalogCoder: Analog Circuit Design via Training-Free Code Generation](https://mlsanity.com/paper/2405.14918/) (2024)
- [Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation](https://mlsanity.com/paper/2308.10335/) (2023)
- [GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Models](https://mlsanity.com/paper/2309.10730/) (2023)
- [ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models](https://mlsanity.com/paper/2406.18770/) (2024)
- [A Survey of Research in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2501.09655/) (2025)
- [AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologies](https://mlsanity.com/paper/2503.00205/) (2025)
- [AmpAgent: An LLM-based Multi-Agent System for Multi-stage Amplifier Schematic Design from Literature for Process and Performance Porting](https://mlsanity.com/paper/2409.14739/) (2024)
- [AMSNet: Netlist Dataset for AMS Circuits](https://mlsanity.com/paper/2405.09045/) (2024)
- [Masala-CHAI: A Large-Scale SPICE Netlist Dataset for Analog Circuits by Harnessing AI](https://mlsanity.com/paper/2411.14299/) (2024)
- [A Survey of Circuit Foundation Model: Foundation AI Models for VLSI Circuit Design and EDA](https://mlsanity.com/paper/2504.03711/) (2025)
- [AMSnet-KG: A Netlist Dataset for LLM-based AMS Circuit Auto-Design Using Knowledge Graph RAG](https://mlsanity.com/paper/2411.13560/) (2024)
- [On the Viability of using LLMs for SW/HW Co-Design: An Example in Designing CiM DNN Accelerators](https://mlsanity.com/paper/2306.06923/) (2023)
- [CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs](https://mlsanity.com/paper/2502.07980/) (2025)

## Built on by (most cited first, 15 of 15)

- [AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMs](https://mlsanity.com/paper/2508.02518/) (2025)
- [MuaLLM: A Multimodal Large Language Model Agent for Circuit Design Assistance with Hybrid Contextual Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.08137/) (2025)
- [Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understanding](https://mlsanity.com/paper/2603.18472/) (2026)
- [CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process](https://mlsanity.com/paper/2509.22339/) (2025)
- [Evaluating LLM-based Workflows for Switched-Mode Power Supply Design](https://mlsanity.com/paper/2507.10639/) (2025)
- [Dataset Construction for Training LLM to Learn Analog Circuit Knowledge](https://mlsanity.com/paper/2508.10409/) (2025)
- [RF-Agent: A Practical Framework for Building Language Agents for RFIC Design](https://mlsanity.com/paper/2607.18772/) (2026)
- [NetlistBench: Evaluating LLM Reliability in SPICE Netlist Recognition and Manipulation](https://mlsanity.com/paper/2608.12197/) (2026)
- [DiagramNet: An End-to-End Recognition Framework and Dataset for Non-Standard System-Level Diagrams](https://mlsanity.com/paper/2605.01338/) (2026)
- [Surveying GenAI-based Automation in Printed Circuit Board Design and Test](https://mlsanity.com/paper/2606.17074/) (2026)
- [PCBnet: A Dataset and Automatic Construction of SPICE Netlists from Schematic Images](https://mlsanity.com/paper/2608.27923/) (2026)
- [Fidelity Is Not Enough: Dispatch-Level Instrumentation for Agentic Datasheet Extraction](https://mlsanity.com/paper/2608.28439/) (2026)
- [Circuit-MLLM: Topological Logic-Guided Latent-Space Visual Reasoning for Circuit Schematic Understanding](https://mlsanity.com/paper/2609.15668/) (2026)
- [Long-Horizon Analog Design Bench: Benchmarking Agents on Hours-Long Analog and Mixed-Signal Circuit Design Tasks](https://mlsanity.com/paper/2609.33356/) (2026)
- [2d-fet-bench: from spatial reasoning to fet design on flakes](https://mlsanity.com/paper/2610.07423/) (2026)

## Abstract (arXiv)

Analog/Mixed-Signal (AMS) circuits play a critical role in the integrated circuit (IC) industry. However, automating Analog/Mixed-Signal (AMS) circuit design has remained a longstanding challenge due to its difficulty and complexity. Although recent advances in Multi-modal Large Language Models (MLLMs) offer promising potential for supporting AMS circuit analysis and design, current research typically evaluates MLLMs on isolated tasks within the domain, lacking a comprehensive benchmark that systematically assesses model capabilities across diverse AMS-related challenges. To address this gap, we introduce AMSbench, a benchmark suite designed to evaluate MLLM performance across critical tasks including circuit schematic perception, circuit analysis, and circuit design. AMSbench comprises approximately 8000 test questions spanning multiple difficulty levels and assesses eight prominent models, encompassing both open-source and proprietary solutions such as Qwen 2.5-VL and Gemini 2.5 Pro. Our evaluation highlights significant limitations in current MLLMs, particularly in complex multi-modal reasoning and sophisticated circuit design tasks. These results underscore the necessity of advancing MLLMs' understanding and effective application of circuit-specific knowledge, thereby narrowing the existing performance gap relative to human expertise and moving toward fully automated AMS circuit design workflows. Our data is released at this URL.