# RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists

arXiv: https://arxiv.org/abs/2608.04783 · HTML: https://mlsanity.com/paper/2608.04783/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 20 of 20)

- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Kimi K2.5: Visual Agentic Intelligence](https://mlsanity.com/paper/2602.02276/) (2026)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://mlsanity.com/paper/2303.03004/) (2023)
- [SWE-QA: Can Language Models Answer Repository-level Code Questions?](https://mlsanity.com/paper/2509.14635/) (2025)
- [RepoQA: Evaluating Long Context Code Understanding](https://mlsanity.com/paper/2406.06025/) (2024)
- [CoReQA: Uncovering Potentials of Language Models in Code Repository Question Answering](https://mlsanity.com/paper/2501.03447/) (2025)
- [LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering](https://mlsanity.com/paper/2509.09614/) (2025)
- [SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring](https://mlsanity.com/paper/2602.03712/) (2026)
- [Hierarchical Knowledge Injection for Improving LLM-based Program Repair](https://mlsanity.com/paper/2506.24015/) (2025)
- [From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level](https://mlsanity.com/paper/2601.03731/) (2026)
- [RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair](https://mlsanity.com/paper/2603.01048/) (2026)
- [GitHub Discussions: An Exploratory Study of Early Adoption](https://mlsanity.com/paper/2102.05230/) (2021)
- [SWE-Adept: An LLM-Based Agentic Framework for Deep Codebase Analysis and Structured Issue Resolution](https://mlsanity.com/paper/2603.01327/) (2026)
- [Agentic Code Reasoning](https://mlsanity.com/paper/2603.01896/) (2026)
- [A Scalable Benchmark for Repository-Oriented Long-Horizon Conversational Context Management](https://mlsanity.com/paper/2603.06358/) (2026)
- [RubberDuckBench: A Benchmark for AI Coding Assistants](https://mlsanity.com/paper/2601.16456/) (2026)

## Built on by (most cited first, 2 of 2)

- [ANTMAN: Adaptive Need Tracking for Multi-Agent Navigation in Large Information Spaces](https://mlsanity.com/paper/2609.33326/) (2026)
- [Coding-Agent Benchmarks Should Match Their Users' Task Flows](https://mlsanity.com/paper/2610.09633/) (2026)

## Abstract (arXiv)

The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing benchmarks largely rely on bug reports from GitHub Issues, which often allow models to bypass genuine understanding via pattern matching on error logs. This misalignment under-measures Edit Bias, which refers to premature generation, where models prematurely propose code modifications instead of understanding the existing repository architecture. Furthermore, current LLM-as-a-Judge scalar scoring suffers from high variance and low interpretability. This work introduces RepoProbe, a novel benchmark for evaluating repository-level code understanding through open-ended Q&A using GitHub Discussions, which focuses on open-ended architectural inquiries rather than defect reporting. To ensure rigorous evaluation, we propose a Checklist-Based Verification Protocol that decomposes answers into atomic, verifiable facts, thereby replacing subjective ratings with objective verification. Our evaluation of state-of-the-art (SOTA) LLMs reveals a persistent gap between high clarity and evidencegrounded technical correctness. It also quantitatively confirms the prevalence of edit bias, in which models prioritize code generation instead of architectural analysis. Finally, we demonstrate that our verification protocol significantly improves evaluation reliability compared to traditional evaluations with scalar scoring.