mlsanityData to 14 Oct 2026

QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture

arXiv 2510.22087 ↗First posted 24 Oct 2025cs.AR cs.AI cs.LGcited by 3

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, remains absent from current large language model (LLM) evaluations. To this end, we present QuArch (pronounced 'quark'), the first benchmark designed to facilitate the development and evaluation of LLM knowledge and reasoning capabilities specifically in computer architecture. QuArch v1.0 provides a comprehensive collection of 2,671 expert-validated question-answer (QA) pairs covering various aspects of computer architecture, including processor design, memory systems, and interconnection networks. Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking in computer architecture. Frontier model accuracies vary widely (from 34% to 73%) on these advanced questions, highlighting persistent gaps in architectural reasoning across analysis, design, and implementation QAs. Furthermore, via fine-tuning we find that QuArch can translate to improved performance on a realistic memory hierarchy design task, resulting in up to 1.99x more area-efficient solutions and up to 40% more viable solutions overall. By holistically assessing fundamental skills, QuArch provides a foundation for building and measuring LLM capabilities that can accelerate innovation in computing systems. The QuArch benchmark and leaderboard are publicly available at: https://quarch.ai/.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 9 of its 87 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

9 of 87 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024CRQBench: A Benchmark of Code Reasoning Questions15 Aug

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  3. 2025Small Language Models are the Future of Agentic AIcited by 270
  4. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  5. 2023ChipGPT: How far are we from natural language hardware designcited by 105
  6. 2025Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verificationcited by 44
  7. 2024Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTScited by 43
  8. 2025SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimizationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Economy of Minds: Emerging Multi-Agent Intelligence with Economic Interactionscited by 5
  2. 2026AgentDSE: Reasoning-Augmented Architectural Design Space Explorationcited by 0
  3. 2026PerfReasoning: How Well Do LLMs Reason on Hardware Performance?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models
  2. 2026PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces
  3. 2026Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning
  4. 2025Can Reasoning Models Reason about Hardware? An Agentic HLS Perspective
  5. 2026WaveformQA: Benchmarking LLM Temporal Reasoning on Digital Waveforms
  6. 2026RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models
  7. 2025QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture
  8. 2025Towards LLM-based Root Cause Analysis of Hardware Design Failures