mlsanityData to 14 Oct 2026

Understanding the Complexity and Its Impact on Testing in ML-Enabled Systems

arXiv 2301.03837 ↗First posted 10 Jan 2023cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) enabled systems are emerging with recent breakthroughs in ML. A model-centric view is widely taken by the literature to focus only on the analysis of ML models. However, only a small body of work takes a system view that looks at how ML components work with the system and how they affect software engineering for MLenabled systems. In this paper, we adopt this system view, and conduct a case study on Rasa 3.0, an industrial dialogue system that has been widely adopted by various companies around the world. Our goal is to characterize the complexity of such a largescale ML-enabled system and to understand the impact of the complexity on testing. Our study reveals practical implications for software engineering for ML-enabled systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 83 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

37 of 83 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  5. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  6. 2019AutoML: A Survey of the State-of-the-Artcited by 390
  7. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
  8. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  9. 2020A Survey on Neural Network Interpretabilitycited by 291
  10. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
  11. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  12. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
25 more
  1. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 124
  2. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  3. 2020Recent Advances and Challenges in Task-oriented Dialog Systemcited by 78
  4. 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 68
  5. 2017Rasa: Open Source Language Understanding and Dialogue Managementcited by 67
  6. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 67
  7. 2021Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Processcited by 56
  8. 2021Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanationscited by 53
  9. 2018Concolic Testing for Deep Neural Networkscited by 51
  10. 2022Causality-based Neural Network Repaircited by 47
  11. 2020Scalable Quantitative Verification For Deep Neural Networkscited by 31
  12. 2020ReluDiff: Differential Verification of Deep Neural Networkscited by 29
  13. 2020NeuroDiff: Scalable Differential Verification of Neural Networks using Fine-Grained Approximationcited by 22
  14. 2021Distribution-Aware Testing of Neural Networks Using Generative Modelscited by 22
  15. 2016On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systemscited by 18
  16. 2021PyCG: Practical Call Graph Generation in Pythoncited by 17
  17. 2019Dialogue Transformerscited by 13
  18. 2021Adapting Software Architectures to Machine Learning Challengescited by 11
  19. 2022Demystifying Dependency Bugs in Deep Learning Stackcited by 9
  20. 2019Operational Calibration: Debugging Confidence Errors for DNNs in the Fieldcited by 6
  21. 2022On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Supportcited by 6
  22. 2018Improving Response Selection in Multi-Turn Dialogue Systems by Incorporating Domain Knowledgecited by 5
  23. 2021Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systemscited by 5
  24. 2021Feature Interactions on Steroids: On the Composition of ML Modelscited by 4
  25. 2019The Dynamics of Software Composition Analysiscited by 1

Built on by

Not cited yet.