# On the Prospects of Dynamic LLM Conversations in Software Development

arXiv: https://arxiv.org/abs/2608.30756 · HTML: https://mlsanity.com/paper/2608.30756/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 33 references that are in the map.
- First posted on arXiv 31 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [Productivity Assessment of Neural Code Completion](https://mlsanity.com/paper/2205.06537/) (2022)
- [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024)
- [Navigating the Complexity of Generative AI Adoption in Software Engineering](https://mlsanity.com/paper/2307.06081/) (2023)
- [Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice](https://mlsanity.com/paper/2404.14901/) (2024)
- [Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation](https://mlsanity.com/paper/2404.16045/) (2024)
- [Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks](https://mlsanity.com/paper/2402.05650/) (2024)
- [Understanding the Role of Large Language Models in Personalizing and Scaffolding Strategies to Combat Academic Procrastination](https://mlsanity.com/paper/2312.13581/) (2023)
- [Self-Prompt Tuning: Enable Autonomous Role-Playing in LLMs](https://mlsanity.com/paper/2407.08995/) (2024)
- [Enhancing LLM-Based Coding Tools through Native Integration of IDE-Derived Static Context](https://mlsanity.com/paper/2402.03630/) (2024)
- [How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering](https://mlsanity.com/paper/2312.11719/) (2023)
- [Developers' Perceptions on the Impact of ChatGPT in Software Development: A Survey](https://mlsanity.com/paper/2405.12195/) (2024)
- [The Impact of AI Tool on Engineering at ANZ Bank An Empirical Study on GitHub Copilot within Corporate Environment](https://mlsanity.com/paper/2402.05636/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have become an essential tool for assisting developers, yet we still lack knowledge on ways to effectively support their interactions during development activities. That is, the quality of interactions with a chat-based LLM still strongly depends on how developers phrase prompts and which information they include. Our goal is to evaluate whether interventions into these interactions with LLMs have an effect on software developers---be it harmful or beneficial. To this end, we conducted a four-month longitudinal study with third-semester computer science students working on a full-stack Web development project using chat-based LLMs under three conditions: (1) a context-aware group received intent-based conversation augmentation, (2) a proactive group received follow-up suggestions and tailored advice, and (3) a control group without intervention. Our augmentations are minimal: (i) to reduce confounding factors and (ii) to isolate treatment effects. Analyzing interaction logs and user surveys revealed no major differences in interaction patterns, indicating no detectable harmful effects in the measured outcomes when intervening in interactions. Moreover, we observed trends of increased satisfaction with the proactive treatment. The results indicate that even with minimal interventions, dynamic guidance mechanisms for developer-LLM interactions show observable effects, such that more severe augmentations may have the potential to substantially improve developer satisfaction.