mlsanityData to 14 Oct 2026

BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems

arXiv 2211.11982 ↗First posted 22 Nov 2022cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We present BotSIM, a data-efficient end-to-end Bot SIMulation toolkit for commercial text-based task-oriented dialog (TOD) systems. BotSIM consists of three major components: 1) a Generator that can infer semantic-level dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing; 2) an agenda-based dialog user Simulator (ABUS) to simulate conversations with the dialog agents; 3) a Remediator to analyze the simulated conversations, visualize the bot health reports and provide actionable remediation suggestions for bot troubleshooting and improvement. We demonstrate BotSIM's effectiveness in end-to-end evaluation, remediation and multi-intent dialog generation via case studies on two commercial bot platforms. BotSIM's "generation-simulation-remediation" paradigm accelerates the end-to-end bot evaluation and iteration process by: 1) reducing manual test cases creation efforts; 2) enabling a holistic gauge of the bot in terms of NLU and end-to-end performance via extensive dialog simulation; 3) improving the bot troubleshooting process with actionable suggestions. A demo of our system can be found at https://tinyurl.com/mryu74cd and a demo video at https://youtu.be/qLi5iSoly30. We have open-sourced the toolkit at https://github.com/salesforce/botsim

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

18 of 25 references.

References · 18

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  5. 2018Building a Conversational Agent Overnight with Dialogue Self-Playcited by 70
  6. 2018Learning Neural Templates for Text Generationcited by 61
  7. 2016A User Simulator for Task-Completion Dialoguescited by 57
  8. 2020ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systemscited by 52
  9. 2019How to Build User Simulators to Train RL-based Dialog Systemscited by 46
  10. 2018Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systemscited by 42
  11. 2016A Sequence-to-Sequence Model for User Simulation in Spoken Dialogue Systemscited by 39
  12. 2018User Modeling for Task Oriented Dialoguescited by 36
6 more
  1. 2020Robustness Testing of Language Understanding in Task-Oriented Dialogcited by 32
  2. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  3. 2022Hierarchical Sketch Induction for Paraphrase Generationcited by 22
  4. 2017End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learningcited by 19
  5. 2021Alexa Conversations: An Extensible Data-driven Approach for Building Task-oriented Dialogue Systemscited by 14
  6. 2017Analysis and Optimization of Convolutional Neural Network Architecturescited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systemscited by 0