# SWE-QA: Can Language Models Answer Repository-level Code Questions?

arXiv: https://arxiv.org/abs/2509.14635 · HTML: https://mlsanity.com/paper/2509.14635/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 86 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Builds on (most cited first, 30 of 31)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation](https://mlsanity.com/paper/2303.12570/) (2023)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph](https://mlsanity.com/paper/2410.14684/) (2024)
- [FineSurE: Fine-grained Summarization Evaluation using LLMs](https://mlsanity.com/paper/2407.00908/) (2024)
- [CoSQA: 20,000+ Web Queries for Code Search and Question Answering](https://mlsanity.com/paper/2105.13239/) (2021)
- [Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning](https://mlsanity.com/paper/2304.11384/) (2023)
- [Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering](https://mlsanity.com/paper/2502.06193/) (2025)
- [SWE-Exp: Experience-Driven Software Issue Resolution](https://mlsanity.com/paper/2507.23361/) (2025)
- [SWE-Debate: Competitive Multi-Agent Debate for Software Issue Resolution](https://mlsanity.com/paper/2507.23348/) (2025)
- [From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging](https://mlsanity.com/paper/2410.01215/) (2024)
- [RepoFusion: Training Code Models to Understand Your Repository](https://mlsanity.com/paper/2306.10998/) (2023)
- [CodeQA: A Question Answering Dataset for Source Code Comprehension](https://mlsanity.com/paper/2109.08365/) (2021)
- [RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation](https://mlsanity.com/paper/2412.17744/) (2024)
- [Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoning](https://mlsanity.com/paper/2508.10848/) (2025)
- [EvoC2Rust: A Skeleton-guided Framework for Project-Level C-to-Rust Translation](https://mlsanity.com/paper/2508.04295/) (2025)
- [CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering](https://mlsanity.com/paper/2501.03447/) (2025)
- [From Code to Courtroom: LLMs as the New Software Judges](https://mlsanity.com/paper/2503.02246/) (2025)
- [ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming](https://mlsanity.com/paper/2505.16667/) (2025)
- [ProCQA: A Large-scale Community-based Programming Question Answering Dataset for Code Search](https://mlsanity.com/paper/2403.16702/) (2024)
- [Beyond Emotion Recognition: A Multi-Turn Multimodal Emotion Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2508.16859/) (2025)
- [CodeQueries: A Dataset of Semantic Queries over Code](https://mlsanity.com/paper/2209.08372/) (2022)
- [CS1QA: A Dataset for Assisting Code-based Question Answering in an Introductory Programming Course](https://mlsanity.com/paper/2210.14494/) (2022)
- [CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering](https://mlsanity.com/paper/2412.14764/) (2024)

## Built on by (most cited first, 30 of 38)

- [SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents](https://mlsanity.com/paper/2601.16746/) (2026)
- [LongCodeZip: Compress Long Context for Code Language Models](https://mlsanity.com/paper/2510.00446/) (2025)
- [SWE-Explore: Benchmarking How Coding Agents Explore Repositories](https://mlsanity.com/paper/2606.07297/) (2026)
- [Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answering](https://mlsanity.com/paper/2601.11255/) (2026)
- [Seeing is Coding: On the Effectiveness of Vision Language Models in Code Understanding](https://mlsanity.com/paper/2602.01785/) (2026)
- [SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding](https://mlsanity.com/paper/2603.16124/) (2026)
- [CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents](https://mlsanity.com/paper/2603.17829/) (2026)
- [EffiSkill: Agent Skill Based Automated Code Efficiency Optimization](https://mlsanity.com/paper/2603.27850/) (2026)
- [The Context Gathering Decision Process: A POMDP Framework for Agentic Search](https://mlsanity.com/paper/2605.07042/) (2026)
- [DeployBench: Benchmarking LLM Agents for Research Artifact Deployment](https://mlsanity.com/paper/2606.05238/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring](https://mlsanity.com/paper/2608.09802/) (2026)
- [FastContext: Training Efficient Repository Explorer for Coding Agents](https://mlsanity.com/paper/2606.14066/) (2026)
- [Fed-SE: Federated Self-Evolution for Privacy-Constrained Multi-Environment LLM Agents](https://mlsanity.com/paper/2512.08870/) (2025)
- [LLM Agents Can See Code Repositories](https://mlsanity.com/paper/2606.14061/) (2026)
- [Repo0: Design-Driven Zero-to-All Code Generation](https://mlsanity.com/paper/2608.19854/) (2026)
- [EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction](https://mlsanity.com/paper/2609.02783/) (2026)
- [HyperAgent: Leveraging Hypergraphs for Topology Optimization in Multi-Agent Communication](https://mlsanity.com/paper/2510.10611/) (2025)
- [ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation](https://mlsanity.com/paper/2604.26923/) (2026)
- [Context Pruning for Coding Agents via Multi-Rubric Latent Reasoning](https://mlsanity.com/paper/2605.15315/) (2026)
- [Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA](https://mlsanity.com/paper/2605.29277/) (2026)
- [SWE-Pruner Pro: The Coder LLM Already Knows What to Prune](https://mlsanity.com/paper/2607.18213/) (2026)
- [RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists](https://mlsanity.com/paper/2608.04783/) (2026)
- [FastCode: Fast and Cost-Efficient Code Understanding and Reasoning](https://mlsanity.com/paper/2603.01012/) (2026)
- [Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry](https://mlsanity.com/paper/2604.10101/) (2026)
- [Neurosymbolic Repo-level Code Localization](https://mlsanity.com/paper/2604.16021/) (2026)
- [Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Study](https://mlsanity.com/paper/2608.01507/) (2026)
- [SWE-Spot: Building Small Repo-Experts with Repository-Centric Learning](https://mlsanity.com/paper/2601.21649/) (2026)
- [ShredBench: Evaluating the Semantic Reasoning Capabilities of Multimodal LLMs in Document Reconstruction](https://mlsanity.com/paper/2604.23813/) (2026)
- [BUILD-AND-FIND: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases](https://mlsanity.com/paper/2605.06136/) (2026)

## Abstract (arXiv)

Understanding and reasoning about entire software repositories is an essential capability for intelligent software engineering tools. While existing benchmarks such as CoSQA and CodeQA have advanced the field, they predominantly focus on small, self-contained code snippets. These setups fail to capture the complexity of real-world repositories, where effective understanding and reasoning often require navigating multiple files, understanding software architecture, and grounding answers in long-range code dependencies. In this paper, we present SWE-QA, a repository-level code question answering (QA) benchmark designed to facilitate research on automated QA systems in realistic code environments. SWE-QA involves 576 high-quality question-answer pairs spanning diverse categories, including intention understanding, cross-file reasoning, and multi-hop dependency analysis. To construct SWE-QA, we first crawled 77,100 GitHub issues from 11 popular repositories. Based on an analysis of naturally occurring developer questions extracted from these issues, we developed a two-level taxonomy of repository-level questions and constructed a set of seed questions for each category. For each category, we manually curated and validated questions and collected their corresponding answers. As a prototype application, we further develop SWE-QA-Agent, an agentic framework in which LLM agents reason and act to find answers automatically. We evaluate six advanced LLMs on SWE-QA under various context augmentation strategies. Experimental results highlight the promise of LLMs, particularly our SWE-QA-Agent framework, in addressing repository-level QA, while also revealing open challenges and pointing to future research directions.