mlsanityData to 14 Oct 2026

SWE-QA: A Dataset and Benchmark for Complex Code Understanding

arXiv 2604.24814 ↗First posted 27 Apr 2026cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce SWE-QA, a text and code corpus aimed at benchmarking multi-hop code comprehension, addressing the gap between simplified evaluation tasks and the complex reasoning required in real-world software development. While existing code understanding benchmarks focus on isolated snippets, developers must routinely connect information across multiple dispersed code segments. The dataset comprises 9,072 multiple-choice questions systematically generated from 12 Python repositories of SWE-bench, evaluating several recurrent reasoning patterns like Declaration-and-Call questions that link entity definitions to their usage, and Interacting-Entity questions that examine the dynamic relationships among multiple collaborating components. Generated through parsing-based entity extraction and Large Language Model assisted question construction with carefully validated distractors, the benchmark distinguishes genuine comprehension from superficial pattern matching. Evaluation of 15 language models (360M to 671B parameters) reveals significant challenges in multi-hop reasoning, with best performance reaching 74.41% accuracy. Dense architectures consistently outperform mixture-of-experts models by 10-14 percentage points, while reasoning-enhanced variants show inconsistent benefits.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 13 of its 17 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

13 of 17 references.

References · 13

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  7. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  8. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  9. 2023Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Codecited by 62
  10. 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 34
  11. 2021CodeQA: A Question Answering Dataset for Source Code Comprehensioncited by 30
  12. 2023Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMscited by 19
1 more
  1. 2023Pitfalls in Language Models for Code Intelligence: A Taxonomy and Surveycited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Deep Agentic Search for Repository-Level Code Question Answering: An Empirical Studycited by 1
  2. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SWE-QA: Can Language Models Answer Repository-level Code Questions?
  2. 2026SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding
  3. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
  4. 2026Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA
  5. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
  6. 2024CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding & Reasoning Capabilities of CodeLLMs
  7. 2025CodeReviewQA: The Code Review Comprehension Assessment for Large Language Models
  8. 2024CRQBench: A Benchmark of Code Reasoning Questions