# A Comparison of Natural Language Understanding Platforms for Chatbots in Software Engineering

arXiv: https://arxiv.org/abs/2012.02640 · HTML: https://mlsanity.com/paper/2012.02640/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [Motivations, Challenges, Best Practices, and Benefits for Bots and Conversational Agents in Software Engineering: A Multivocal Literature Review](https://mlsanity.com/paper/2409.11864/) (2024)
- [Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education](https://mlsanity.com/paper/2309.13059/) (2023)
- [Introducing Brain-like Concepts to Embodied Hand-crafted Dialog Management System](https://mlsanity.com/paper/2406.08996/) (2024)
- [LSTM-RASA Based Agri Farm Assistant for Farmers](https://mlsanity.com/paper/2204.09717/) (2022)
- [From Human-to-Human to Human-to-Bot Conversations in Software Engineering](https://mlsanity.com/paper/2405.12712/) (2024)
- [Synergizing LLMs and Knowledge Graphs: A Novel Approach to Software Repository-Related Question Answering](https://mlsanity.com/paper/2412.03815/) (2024)
- [A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback](https://mlsanity.com/paper/2605.18073/) (2026)
- [Creation Of A ChatBot Based On Natural Language Proccesing For Whatsapp](https://mlsanity.com/paper/2310.10675/) (2023)
- [An Approach for Auto Generation of Labeling Functions for Software Engineering Chatbots](https://mlsanity.com/paper/2410.07094/) (2024)
- [Enhancing Software Development with Context-Aware Conversational Agents: A User Study on Developer Interactions with Chatbots](https://mlsanity.com/paper/2505.08648/) (2025)

## Abstract (arXiv)

Chatbots are envisioned to dramatically change the future of Software Engineering, allowing practitioners to chat and inquire about their software projects and interact with different services using natural language. At the heart of every chatbot is a Natural Language Understanding (NLU) component that enables the chatbot to understand natural language input. Recently, many NLU platforms were provided to serve as an off-the-shelf NLU component for chatbots, however, selecting the best NLU for Software Engineering chatbots remains an open challenge. Therefore, in this paper, we evaluate four of the most commonly used NLUs, namely IBM Watson, Google Dialogflow, Rasa, and Microsoft LUIS to shed light on which NLU should be used in Software Engineering based chatbots. Specifically, we examine the NLUs' performance in classifying intents, confidence scores stability, and extracting entities. To evaluate the NLUs, we use two datasets that reflect two common tasks performed by Software Engineering practitioners, 1) the task of chatting with the chatbot to ask questions about software repositories 2) the task of asking development questions on Q&A forums (e.g., Stack Overflow). According to our findings, IBM Watson is the best performing NLU when considering the three aspects (intents classification, confidence scores, and entity extraction). However, the results from each individual aspect show that, in intents classification, IBM Watson performs the best with an F1-measure > 84%, but in confidence scores, Rasa comes on top with a median confidence score higher than 0.91. Our results also show that all NLUs, except for Dialogflow, generally provide trustable confidence scores. For entity extraction, Microsoft LUIS and IBM Watson outperform other NLUs in the two SE tasks. Our results provide guidance to software engineering practitioners when deciding which NLU to use in their chatbots.