# From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering

arXiv: https://arxiv.org/abs/2512.23844 · HTML: https://mlsanity.com/paper/2512.23844/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 29 Dec 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 20 of 20)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction](https://mlsanity.com/paper/2209.11515/) (2022)
- [SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?](https://mlsanity.com/paper/2502.12115/) (2025)
- [TRAIL: Trace Reasoning and Agentic Issue Localization](https://mlsanity.com/paper/2505.08638/) (2025)
- [Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories](https://mlsanity.com/paper/2506.18824/) (2025)
- [Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors](https://mlsanity.com/paper/2412.09416/) (2024)
- [Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach](https://mlsanity.com/paper/2407.12687/) (2024)
- [Evaluating Agent-based Program Repair at Google](https://mlsanity.com/paper/2501.07531/) (2025)
- [An Empirical Study on LLM-based Agents for Automated Bug Fixing](https://mlsanity.com/paper/2411.10213/) (2024)
- [EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework](https://mlsanity.com/paper/2504.14928/) (2025)
- [ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments](https://mlsanity.com/paper/2502.19852/) (2025)
- [Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models](https://mlsanity.com/paper/2407.11470/) (2024)
- [Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summaries](https://mlsanity.com/paper/2409.00844/) (2024)
- [How is Google using AI for internal code migrations?](https://mlsanity.com/paper/2501.06972/) (2025)
- [Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassian](https://mlsanity.com/paper/2501.11264/) (2025)

## Built on by (most cited first, 2 of 2)

- [CollabSkill: Evaluating Human-Agent Collaboration On Real-World Tasks](https://mlsanity.com/paper/2606.09833/) (2026)
- [Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents](https://mlsanity.com/paper/2607.23670/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) evolve from code generators into collaborative partners for software engineers, our methods for evaluation are lagging. Current benchmarks, focused on code correctness, fail to capture the nuanced, interactive behaviors essential for successful human-AI partnership. To bridge this evaluation gap, this paper makes two core contributions. First, we present a foundational taxonomy of desirable agent behaviors for enterprise software engineering, derived from an analysis of 91 sets of user-defined agent rules. This taxonomy defines four key expectations of agent behavior: Adhere to Standards and Processes, Ensure Code Quality and Reliability, Solving Problems Effectively, and Collaborating with the User. Second, recognizing that these expectations are not static, we introduce the Context-Adaptive Behavior (CAB) Framework. This emerging framework reveals how behavioral expectations shift along two empirically-derived axes: the Time Horizon (from immediate needs to future ideals), established through interviews with 15 expert engineers, and the Type of Work (from enterprise production to rapid prototyping, for example), identified through a prompt analysis of a prototyping agent. Together, these contributions offer a human-centered foundation for designing and evaluating the next generation of AI agents, moving the field's focus from the correctness of generated code toward the dynamics of true collaborative intelligence.