mlsanityData to 7 Oct 2026

QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture

arXiv 2501.01892 ↗First posted 3 Jan 2025cs.AR cs.AI cs.LGcited by 4

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

We introduce QuArch, a dataset of 1500 human-validated question-answer pairs designed to evaluate and enhance language models' understanding of computer architecture. The dataset covers areas including processor design, memory systems, and performance optimization. Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%. We observe notable struggles in memory systems, interconnection networks, and benchmarking. Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research. The dataset and leaderboard are at https://harvard-edge.github.io/QuArch/.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 10 of its 16 references that are in the map. First posted on arXiv 3 Jan 2025.

Builds on

10 of 16 references.

References · 10

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  6. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  7. 2021Examining Zero-Shot Vulnerability Repair with Large Language Modelscited by 105
  8. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  9. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  10. 2023nl2spec: Interactively Translating Unstructured Natural Language to Temporal Logics with Large Language Modelscited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimizationcited by 11
  2. 2026Computer Architecture's AlphaZero Moment: Automated Discovery in an Encircled Worldcited by 4
  3. 2026CacheMind: From Miss Rates to Why -- Natural-Language, Trace-Grounded Reasoning for Cache Replacementcited by 0
  4. 2026CAKE: Cloud Architecture Knowledge Evaluation of Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture
  2. 2025AirQA: A Comprehensive QA Dataset for AI Research with Instance-Level Evaluation
  3. 2026CAFE: A Compound-AI Factorial Evaluation Framework
  4. 2026AgentHop: A Diagnostic Benchmark for Agentic Multi-Hop Scientific Question Answering
  5. 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
  6. 2025Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasets
  7. 2026DAREBench: Deployment-Aware and Reliable Evaluation of Models as Agents
  8. 2026Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings