# End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics

arXiv: https://arxiv.org/abs/2211.03511 · HTML: https://mlsanity.com/paper/2211.03511/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 86 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [A Network-based End-to-End Trainable Task-oriented Dialogue System](https://mlsanity.com/paper/1604.04562/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [A Survey on Dialogue Systems: Recent Advances and New Frontiers](https://mlsanity.com/paper/1711.01731/) (2017)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue](https://mlsanity.com/paper/2009.13570/) (2020)
- [Rasa: Open Source Language Understanding and Dialogue Management](https://mlsanity.com/paper/1712.05181/) (2017)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education](https://mlsanity.com/paper/2106.07340/) (2021)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [DIET: Lightweight Language Understanding for Dialogue Systems](https://mlsanity.com/paper/2004.09936/) (2020)
- [Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems](https://mlsanity.com/paper/1511.06931/) (2015)
- [A Survey of Available Corpora for Building Data-Driven Dialogue Systems](https://mlsanity.com/paper/1512.05742/) (2015)
- [Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning](https://mlsanity.com/paper/1606.02560/) (2016)
- [StarSpace: Embed All The Things!](https://mlsanity.com/paper/1709.03856/) (2017)
- [Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript](https://mlsanity.com/paper/2102.00804/) (2021)
- [End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.10712/) (2017)
- [Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management](https://mlsanity.com/paper/1707.00130/) (2017)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)
- [Dialogue Transformers](https://mlsanity.com/paper/1910.00486/) (2019)
- [Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational AI: HERMIT NLU](https://mlsanity.com/paper/1910.00912/) (2019)
- [N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypotheses](https://mlsanity.com/paper/2106.06519/) (2021)
- [Automatic Generation of Headlines for Online Math Questions](https://mlsanity.com/paper/1912.00839/) (2019)
- [Natural Language Interactions in Autonomous Vehicles: Intent Detection and Slot Filling from Passenger Utterances](https://mlsanity.com/paper/1904.10500/) (2019)

## Built on by (most cited first, 2 of 2)

- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home](https://mlsanity.com/paper/2306.00482/) (2023)

## Abstract (arXiv)

The advances in language-based Artificial Intelligence (AI) technologies applied to build educational applications can present AI for social-good opportunities with a broader positive impact. Across many disciplines, enhancing the quality of mathematics education is crucial in building critical thinking and problem-solving skills at younger ages. Conversational AI systems have started maturing to a point where they could play a significant role in helping students learn fundamental math concepts. This work presents a task-oriented Spoken Dialogue System (SDS) built to support play-based learning of basic math concepts for early childhood education. The system has been evaluated via real-world deployments at school while the students are practicing early math concepts with multimodal interactions. We discuss our efforts to improve the SDS pipeline built for math learning, for which we explore utilizing MathBERT representations for potential enhancement to the Natural Language Understanding (NLU) module. We perform an end-to-end evaluation using real-world deployment outputs from the Automatic Speech Recognition (ASR), Intent Recognition, and Dialogue Manager (DM) components to understand how error propagation affects the overall performance in real-world scenarios.