# CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance

arXiv: https://arxiv.org/abs/2507.10646 · HTML: https://mlsanity.com/paper/2507.10646/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 14 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents](https://mlsanity.com/paper/2504.08703/) (2025)
- [LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical Evaluation](https://mlsanity.com/paper/2404.10100/) (2024)
- [BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues](https://mlsanity.com/paper/2310.13650/) (2023)
- [Leveraging Large Language Models to Improve REST API Testing](https://mlsanity.com/paper/2312.00894/) (2023)
- [StackEval: Benchmarking LLMs in Coding Assistance](https://mlsanity.com/paper/2412.05288/) (2024)
- [ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments](https://mlsanity.com/paper/2502.19852/) (2025)
- [Automated Test Generation for REST APIs: No Time to Rest Yet](https://mlsanity.com/paper/2204.08348/) (2022)
- [InfiBench: Evaluating the Question-Answering Capabilities of Code Large Language Models](https://mlsanity.com/paper/2404.07940/) (2024)
- [A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs](https://mlsanity.com/paper/2411.07098/) (2024)

## Built on by (most cited first, 5 of 5)

- [CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents](https://mlsanity.com/paper/2603.17829/) (2026)
- [CODESTRUCT: Code Agents over Structured Action Spaces](https://mlsanity.com/paper/2604.05407/) (2026)
- [Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects](https://mlsanity.com/paper/2604.06373/) (2026)
- [Out of Sight: Compression-Aware Content Protection against Agentic Crawlers](https://mlsanity.com/paper/2607.08180/) (2026)
- [CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMs](https://mlsanity.com/paper/2608.00018/) (2026)

## Abstract (arXiv)

Programming assistants powered by large language models have improved dramatically, yet existing benchmarks still evaluate them in narrow code-generation settings. Recent efforts such as InfiBench and StackEval rely on Stack Overflow questions and remain limited to single-turn interactions, manually curated data, and isolated snippets rather than full project environments. We introduce CodeAssistBench (CAB), the first benchmark for evaluating multi-turn, project-grounded programming assistance at scale. CAB automatically constructs datasets from GitHub issues tagged as questions, using an LLM-driven pipeline that filters noise, extracts runnable contexts, builds executable containers, and verifies environment correctness. This enables continuous, automated expansion across diverse repositories without manual intervention. Using CAB, we create a testbed of 3,286 real-world issues across 214 repositories, spanning seven languages. Evaluating state-of-the-art models reveals a substantial gap: while models achieve 70-83% accuracy on Stack Overflow-style questions, they solve only 7.22-16.49% of CAB issues from post-training-cutoff repositories. These results highlight a fundamental challenge: current LLMs struggle to provide assistance in realistic, project-specific contexts despite strong performance on traditional Q&A benchmarks. CAB provides a scalable, reproducible framework for advancing research in multi-turn, codebase-grounded programming agents. The benchmark and pipeline are fully automated and publicly available at https://github.com/amazon-science/CodeAssistBench/.