# NLU for Game-based Learning in Real: Initial Evaluations

arXiv: https://arxiv.org/abs/2205.13754 · HTML: https://mlsanity.com/paper/2205.13754/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 70 references that are in the map.
- First posted on arXiv 27 May 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Benchmarking Natural Language Understanding Services for building Conversational Agents](https://mlsanity.com/paper/1903.05566/) (2019)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Learning End-to-End Goal-Oriented Dialog](https://mlsanity.com/paper/1605.07683/) (2016)
- [ConveRT: Efficient and Accurate Conversational Representations from Transformers](https://mlsanity.com/paper/1911.03688/) (2019)
- [Rasa: Open Source Language Understanding and Dialogue Management](https://mlsanity.com/paper/1712.05181/) (2017)
- [Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access](https://mlsanity.com/paper/1609.00777/) (2016)
- [DIET: Lightweight Language Understanding for Dialogue Systems](https://mlsanity.com/paper/2004.09936/) (2020)
- [Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems](https://mlsanity.com/paper/1511.06931/) (2015)
- [StarSpace: Embed All The Things!](https://mlsanity.com/paper/1709.03856/) (2017)
- [End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning](https://mlsanity.com/paper/1711.10712/) (2017)
- [Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management](https://mlsanity.com/paper/1707.00130/) (2017)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)
- [Hierarchical Multi-Task Natural Language Understanding for Cross-domain Conversational AI: HERMIT NLU](https://mlsanity.com/paper/1910.00912/) (2019)
- [Natural Language Interactions in Autonomous Vehicles: Intent Detection and Slot Filling from Passenger Utterances](https://mlsanity.com/paper/1904.10500/) (2019)
- [SimpleDS: A Simple Deep Reinforcement Learning Dialogue System](https://mlsanity.com/paper/1601.04574/) (2016)
- [An Exploratory Study of Argumentative Writing by Young Students: A Transformer-based Approach](https://mlsanity.com/paper/2006.09873/) (2020)
- [Modeling Intent, Dialog Policies and Response Adaptation for Goal-Oriented Interactions](https://mlsanity.com/paper/1912.10130/) (2019)
- [Semi-supervised Interactive Intent Labeling](https://mlsanity.com/paper/2104.13406/) (2021)
- [Audio-Visual Understanding of Passenger Intents for In-Cabin Conversational Agents](https://mlsanity.com/paper/2007.03876/) (2020)

## Built on by (most cited first, 2 of 2)

- [End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics](https://mlsanity.com/paper/2211.03511/) (2022)
- [Inspecting Spoken Language Understanding from Kids for Basic Math Learning at Home](https://mlsanity.com/paper/2306.00482/) (2023)

## Abstract (arXiv)

Intelligent systems designed for play-based interactions should be contextually aware of the users and their surroundings. Spoken Dialogue Systems (SDS) are critical for these interactive agents to carry out effective goal-oriented communication with users in real-time. For the real-world (i.e., in-the-wild) deployment of such conversational agents, improving the Natural Language Understanding (NLU) module of the goal-oriented SDS pipeline is crucial, especially with limited task-specific datasets. This study explores the potential benefits of a recently proposed transformer-based multi-task NLU architecture, mainly to perform Intent Recognition on small-size domain-specific educational game datasets. The evaluation datasets were collected from children practicing basic math concepts via play-based interactions in game-based learning settings. We investigate the NLU performances on the initial proof-of-concept game datasets versus the real-world deployment datasets and observe anticipated performance drops in-the-wild. We have shown that compared to the more straightforward baseline approaches, Dual Intent and Entity Transformer (DIET) architecture is robust enough to handle real-world data to a large extent for the Intent Recognition task on these domain-specific in-the-wild game datasets.