# Developer-LLM Conversations: An Empirical Study of Interactions and Generated Code Quality

arXiv: https://arxiv.org/abs/2509.10402 · HTML: https://mlsanity.com/paper/2509.10402/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Sep 2025.

## Built on by (most cited first, 6 of 6)

- [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://mlsanity.com/paper/2606.29957/) (2026)
- [From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality](https://mlsanity.com/paper/2607.13196/) (2026)
- [AdaptAgent: A Multi-agent, Domain-Guided Reasoning Framework for Code Adaptation](https://mlsanity.com/paper/2608.04459/) (2026)
- [Do Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code Generation](https://mlsanity.com/paper/2608.11513/) (2026)
- [EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models](https://mlsanity.com/paper/2609.01611/) (2026)
- [SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction](https://mlsanity.com/paper/2610.11559/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used by developers for conversational coding assistance, including code generation, technical questions, and iterative problem solving. However, limited empirical evidence exists on the topics developers discuss with LLMs, how interaction patterns vary across tasks, and how the quality of generated code changes across conversational turns. To address this, we construct CodeChat, a large-scale dataset derived from WildChat. CodeChat contains 587,568 real-world developer-LLM conversations and 1.7 million code snippets across more than 20 programming languages. We identify the topics developers raise and examine how their engagement varies across topics. We then assess the quality of generated code across five programming languages (i.e., Python, JavaScript, C++, Java, and C#) and analyze how quality issues evolve across conversational turns. We find that developers most frequently seek help with web design (9.6% of conversations) and machine learning (8.7%), and that conversations often grow long because developers repeatedly shift use cases. LLM-generated code exhibits prevalent, language-specific issues whose prevalence does not consistently decrease in interactions lasting at least five turns. Among successful syntax-error resolutions, prompts that explicitly identify the error and request a fix are the most frequently observed. Our findings suggest that conversational assistants should track evolving developer intent and monitor code quality across turns.