# Agentic Repository Mining: A Multi-Task Evaluation

arXiv: https://arxiv.org/abs/2605.04845 · HTML: https://mlsanity.com/paper/2605.04845/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 6 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 14 of 14)

- [Using simulation studies to evaluate statistical methods](https://mlsanity.com/paper/1712.03198/) (2017)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [From Commit Message Generation to History-Aware Commit Message Completion](https://mlsanity.com/paper/2308.07655/) (2023)
- [Improving Vulnerability Inspection Efficiency Using Active Learning](https://mlsanity.com/paper/1803.06545/) (2018)
- [Boosting Automatic Commit Classification Into Maintenance Activities By Utilizing Source Code Changes](https://mlsanity.com/paper/1711.05340/) (2017)
- [Evaluating Large Language Models for Code Review](https://mlsanity.com/paper/2505.20206/) (2025)
- [Promises, Perils, and (Timely) Heuristics for Mining Coding Agent Activity](https://mlsanity.com/paper/2601.18345/) (2026)
- [Are Coding Agents Generating Over-Mocked Tests? An Empirical Study](https://mlsanity.com/paper/2602.00409/) (2026)
- [Impact assessment for vulnerabilities in open-source software libraries](https://mlsanity.com/paper/1504.04971/) (2015)
- [Security Defect Detection via Code Review: A Study of the OpenStack and Qt Communities](https://mlsanity.com/paper/2307.02326/) (2023)
- [LLM-Based Detection of Tangled Code Changes for Higher-Quality Method-Level Bug Datasets](https://mlsanity.com/paper/2505.08263/) (2025)
- [Are We All Using Agents the Same Way? An Empirical Study of Core and Peripheral Developers Use of Coding Agents](https://mlsanity.com/paper/2601.20106/) (2026)
- [Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study](https://mlsanity.com/paper/2506.11561/) (2025)
- [SHERPA: A Model-Driven Framework for Large Language Model Execution](https://mlsanity.com/paper/2509.00272/) (2025)

## Abstract (arXiv)

Mining software repositories often requires classifying artifacts like commits, reviews, code lines, or entire repositories into categories. Human labeling is expensive and error-prone; limited context frequently leads to misclassifications or uncertainty in labels. We investigate whether LLM agents that dynamically explore repositories through standard bash commands can match the classification quality of simple LLMs that receive pre-engineered context. Across four tasks, eight approach configurations, and 4943 classifications, agents achieve competitive accuracy despite retrieving their own context. The primary advantage is robustness: agents avoid context-window overflows and scale independently of artifact size. A manual diagnosis of 100 cases where approaches disagree with the ground truth reveals specification ambiguities and labels produced under limited context, suggesting that accuracy against such ground truth may underestimate approaches with broader context access.