# QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture

arXiv: https://arxiv.org/abs/2501.01892 · HTML: https://mlsanity.com/paper/2501.01892/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 3 Jan 2025.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 10 of 10)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems](https://mlsanity.com/paper/2010.06028/) (2020)
- [nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Models](https://mlsanity.com/paper/2303.04864/) (2023)

## Built on by (most cited first, 4 of 4)

- [SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimization](https://mlsanity.com/paper/2508.20258/) (2025)
- [Computer Architecture's AlphaZero Moment: Automated Discovery in an Encircled World](https://mlsanity.com/paper/2604.03312/) (2026)
- [CacheMind: From Miss Rates to Why -- Natural-Language, Trace-Grounded Reasoning for Cache Replacement](https://mlsanity.com/paper/2602.12422/) (2026)
- [CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models](https://mlsanity.com/paper/2604.05755/) (2026)

## Abstract (arXiv)

We introduce QuArch, a dataset of 1500 human-validated question-answer pairs designed to evaluate and enhance language models' understanding of computer architecture. The dataset covers areas including processor design, memory systems, and performance optimization. Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%. We observe notable struggles in memory systems, interconnection networks, and benchmarking. Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research. The dataset and leaderboard are at https://harvard-edge.github.io/QuArch/.