mlsanityData to 7 Oct 2026

Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review

arXiv 2409.11864 ↗First posted 18 Sep 2024cs.SEcited by 6

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Bots are software systems designed to support users by automating a specific process, task, or activity. When such systems implement a conversational component to interact with the users, they are also known as conversational agents. Bots, particularly in their conversation-oriented version and AI-powered, have seen their adoption increase over time for software development and engineering purposes. Despite their exciting potential, ulteriorly enhanced by the advent of Generative AI and Large Language Models, bots still need to be improved to develop and integrate into the development cycle since practitioners report that bots add additional challenges that may worsen rather than improve. In this work, we aim to provide a taxonomy for characterizing bots, as well as a series of challenges for their adoption for Software Engineering associated with potential mitigation strategies. To reach our objectives, we conducted a multivocal literature review, reviewing both research and practitioner's literature. Through such an approach, we hope to contribute to both researchers and practitioners by providing first, a series of future research routes to follow, second, a list of strategies to adopt for improving the use of bots for software engineering purposes, and third, enforce a technology and knowledge transfer from the research field to the practitioners one, that is one of the primary goal of multivocal literature reviews.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 166 references that are in the map. First posted on arXiv 18 Sep 2024.

Builds on

23 of 166 references.

References · 23

  1. 2017Guidelines for including grey literature and conducting multivocal literature reviews in software engineeringcited by 38
  2. 2023Towards Human-Bot Collaborative Software Architecting with ChatGPTcited by 31
  3. 2022Automating Dependency Updates in Practice: An Exploratory Study on GitHub Dependabotcited by 12
  4. 2021Software-Based Dialogue Systems: Survey, Taxonomy and Challengescited by 11
  5. 2018How to Design a Program Repair Bot? Insights from the Repairnator Projectcited by 10
  6. 2020Detecting and Characterizing Bots that Commit Codecited by 10
  7. 2020A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineeringcited by 10
  8. 2020A Multi-Vocal Review of Security Orchestrationcited by 8
  9. 2020An Empirical Study of Bots in Software Development -- Characteristics and Challenges from a Practitioner's Perspectivecited by 8
  10. 2021Quality Gatekeepers: Investigating the Effects ofCode Review Bots on Pull Request Activitiescited by 8
  11. 2021Don't Disturb Me: Challenges of Interacting with SoftwareBots on Open Source Software Projectscited by 8
  12. 2021Bots Don't Mind Waiting, Do They? Comparing the Interaction With Automatically and Manually Created Pull Requestscited by 5
11 more
  1. 2019Explainable Software Bot Contributions: Case Study of Automated Bug Fixescited by 4
  2. 2016Control-Flow Integrity: Precision, Security, and Performancecited by 3
  3. 2017Programming Bots by Synthesizing Natural Language Expressions into API Invocationscited by 3
  4. 2020Challenges and guidelines on designing test cases for test botscited by 2
  5. 2022Together or Apart? Investigating a mediator bot to aggregate bot's comments on pull requestscited by 2
  6. 2019An Additional Set of (Automated) Eyes: Chatbots for Agile Retrospectivescited by 1
  7. 2020Perception and Acceptance of an Autonomous Refactoring Botcited by 1
  8. 2020An Exploratory Study of Bot Commitscited by 1
  9. 2021Designing a Bot for Efficient Distribution of Service Requestscited by 1
  10. 2021Do Bots Modify the Workflow of GitHub Teams?cited by 1
  11. 2023Optimizing Workflow for Elite Developers: Perspectives on Leveraging SE Botscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Software Development with Context-Aware Conversational Agents: A User Study on Developer Interactions with Chatbots
  2. 2023DevBots can co-design APIs
  3. 2026Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
  4. 2025From Correctness to Collaboration: Toward a Human-Centered Framework for Evaluating AI Agent Behavior in Software Engineering
  5. 2025CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
  6. 2026Software as Content: Dynamic Applications as the Human-Agent Interaction Layer ↗
  7. 2025Generative AI and the Transformation of Software Development Practices
  8. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead