# Quality Assurance of Generative Dialog Models in an Evolving Conversational Agent Used for Swedish Language Practice

arXiv: https://arxiv.org/abs/2203.15414 · HTML: https://mlsanity.com/paper/2203.15414/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 57 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation](https://mlsanity.com/paper/2109.06379/) (2021)
- [Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry](https://mlsanity.com/paper/1812.05389/) (2018)
- [Symbolic, Distributed and Distributional Representations for Natural Language Processing in the Era of Deep Learning: a Survey](https://mlsanity.com/paper/1702.00764/) (2017)
- [The AIQ Meta-Testbed: Pragmatically Bridging Academic AI Testing and Industrial Q Needs](https://mlsanity.com/paper/2009.05260/) (2020)
- [Agility in Software 2.0 -- Notebook Interfaces and MLOps with Buttresses and Rebars](https://mlsanity.com/paper/2111.14142/) (2021)
- [Benefitting from the Grey Literature in Software Engineering Research](https://mlsanity.com/paper/1911.12038/) (2019)
- [Benchmarking as Empirical Standard in Software Engineering Research](https://mlsanity.com/paper/2105.00272/) (2021)

## Abstract (arXiv)

Due to the migration megatrend, efficient and effective second-language acquisition is vital. One proposed solution involves AI-enabled conversational agents for person-centered interactive language practice. We present results from ongoing action research targeting quality assurance of proprietary generative dialog models trained for virtual job interviews. The action team elicited a set of 38 requirements for which we designed corresponding automated test cases for 15 of particular interest to the evolving solution. Our results show that six of the test case designs can detect meaningful differences between candidate models. While quality assurance of natural language processing applications is complex, we provide initial steps toward an automated framework for machine learning model selection in the context of an evolving conversational agent. Future work will focus on model selection in an MLOps setting.