# Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review

arXiv: https://arxiv.org/abs/2409.11864 · HTML: https://mlsanity.com/paper/2409.11864/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 166 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 23 of 23)

- [Guidelines for including grey literature and conducting multivocal literature reviews in software engineering](https://mlsanity.com/paper/1707.02553/) (2017)
- [Towards Human-Bot Collaborative Software Architecting with ChatGPT](https://mlsanity.com/paper/2302.14600/) (2023)
- [Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabot](https://mlsanity.com/paper/2206.07230/) (2022)
- [How to Design a Program Repair Bot? Insights from the Repairnator Project](https://mlsanity.com/paper/1811.09852/) (2018)
- [Software-Based Dialogue Systems: Survey, Taxonomy and Challenges](https://mlsanity.com/paper/2106.10901/) (2021)
- [Detecting and Characterizing Bots that Commit Code](https://mlsanity.com/paper/2003.03172/) (2020)
- [A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering](https://mlsanity.com/paper/2012.02640/) (2020)
- [A Multi-Vocal Review of Security Orchestration](https://mlsanity.com/paper/2002.09190/) (2020)
- [An Empirical Study of Bots in Software Development -- Characteristics and Challenges from a Practitioner's Perspective](https://mlsanity.com/paper/2005.13969/) (2020)
- [Quality Gatekeepers: Investigating the Effects ofCode Review Bots on Pull Request Activities](https://mlsanity.com/paper/2103.13547/) (2021)
- [Don't Disturb Me: Challenges of Interacting with SoftwareBots on Open Source Software Projects](https://mlsanity.com/paper/2103.13950/) (2021)
- [Bots Don't Mind Waiting, Do They? Comparing the Interaction With Automatically and Manually Created Pull Requests](https://mlsanity.com/paper/2103.03591/) (2021)
- [Explainable Software Bot Contributions: Case Study of Automated Bug Fixes](https://mlsanity.com/paper/1905.02597/) (2019)
- [Control-Flow Integrity: Precision, Security, and Performance](https://mlsanity.com/paper/1602.04056/) (2016)
- [Programming Bots by Synthesizing Natural Language Expressions into API Invocations](https://mlsanity.com/paper/1711.05410/) (2017)
- [Challenges and guidelines on designing test cases for test bots](https://mlsanity.com/paper/2004.10143/) (2020)
- [Together or Apart? Investigating a mediator bot to aggregate bot's comments on pull requests](https://mlsanity.com/paper/2208.01624/) (2022)
- [An Additional Set of (Automated) Eyes: Chatbots for Agile Retrospectives](https://mlsanity.com/paper/1903.02443/) (2019)
- [Perception and Acceptance of an Autonomous Refactoring Bot](https://mlsanity.com/paper/2001.02553/) (2020)
- [An Exploratory Study of Bot Commits](https://mlsanity.com/paper/2003.07961/) (2020)
- [Designing a Bot for Efficient Distribution of Service Requests](https://mlsanity.com/paper/2103.05970/) (2021)
- [Do Bots Modify the Workflow of GitHub Teams?](https://mlsanity.com/paper/2103.09319/) (2021)
- [Optimizing Workflow for Elite Developers: Perspectives on Leveraging SE Bots](https://mlsanity.com/paper/2304.14828/) (2023)

## Built on by (most cited first, 6 of 6)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025)
- [Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projects](https://mlsanity.com/paper/2511.09268/) (2025)
- [From Junior to Senior: Allocating Agency and Navigating Professional Growth in Agentic AI-Mediated Software Engineering](https://mlsanity.com/paper/2602.00496/) (2026)
- [Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsing](https://mlsanity.com/paper/2511.12630/) (2025)
- [FLARE: Agentic Coverage-Guided Fuzzing for LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2604.05289/) (2026)

## Abstract (arXiv)

Bots are software systems designed to support users by automating a specific process, task, or activity. When such systems implement a conversational component to interact with the users, they are also known as conversational agents. Bots, particularly in their conversation-oriented version and AI-powered, have seen their adoption increase over time for software development and engineering purposes. Despite their exciting potential, ulteriorly enhanced by the advent of Generative AI and Large Language Models, bots still need to be improved to develop and integrate into the development cycle since practitioners report that bots add additional challenges that may worsen rather than improve. In this work, we aim to provide a taxonomy for characterizing bots, as well as a series of challenges for their adoption for Software Engineering associated with potential mitigation strategies. To reach our objectives, we conducted a multivocal literature review, reviewing both research and practitioner's literature. Through such an approach, we hope to contribute to both researchers and practitioners by providing first, a series of future research routes to follow, second, a list of strategies to adopt for improving the use of bots for software engineering purposes, and third, enforce a technology and knowledge transfer from the research field to the practitioners one, that is one of the primary goal of multivocal literature reviews.