# Learning to generate and corr- uh I mean repair language in real-time

arXiv: https://arxiv.org/abs/2308.11683 · HTML: https://mlsanity.com/paper/2308.11683/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 69 references that are in the map.
- First posted on arXiv 22 Aug 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)
- [Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features](https://mlsanity.com/paper/2107.06963/) (2021)
- [Compositional Generalization in Image Captioning](https://mlsanity.com/paper/1909.04402/) (2019)
- [Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU](https://mlsanity.com/paper/2109.07364/) (2021)
- [Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU](https://mlsanity.com/paper/2010.05330/) (2020)
- [Grammar-Based Grounded Lexicon Learning](https://mlsanity.com/paper/2202.08806/) (2022)
- [TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model](https://mlsanity.com/paper/2305.10845/) (2023)
- [Re-framing Incremental Deep Language Models for Dialogue Processing with Multi-task Learning](https://mlsanity.com/paper/2011.06754/) (2020)
- [Bootstrapping incremental dialogue systems from minimal data: the generalisation power of dialogue grammars](https://mlsanity.com/paper/1709.07858/) (2017)
- [Learning how to learn: an adaptive dialogue agent for incrementally learning visually grounded word meanings](https://mlsanity.com/paper/1709.10423/) (2017)
- [Bootstrapping incremental dialogue systems: using linguistic knowledge to learn from minimal data](https://mlsanity.com/paper/1612.00347/) (2016)
- [Training an adaptive dialogue policy for interactive learning of visually grounded word meanings](https://mlsanity.com/paper/1709.10426/) (2017)

## Built on by (most cited first, 1 of 1)

- [Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models](https://mlsanity.com/paper/2409.14247/) (2024)

## Abstract (arXiv)

In conversation, speakers produce language incrementally, word by word, while continuously monitoring the appropriateness of their own contribution in the dynamically unfolding context of the conversation; and this often leads them to repair their own utterance on the fly. This real-time language processing capacity is furthermore crucial to the development of fluent and natural conversational AI. In this paper, we use a previously learned Dynamic Syntax grammar and the CHILDES corpus to develop, train and evaluate a probabilistic model for incremental generation where input to the model is a purely semantic generation goal concept in Type Theory with Records (TTR). We show that the model's output exactly matches the gold candidate in 78% of cases with a ROUGE-l score of 0.86. We further do a zero-shot evaluation of the ability of the same model to generate self-repairs when the generation goal changes mid-utterance. Automatic evaluation shows that the model can generate self-repairs correctly in 85% of cases. A small human evaluation confirms the naturalness and grammaticality of the generated self-repairs. Overall, these results further highlight the generalisation power of grammar-based models and lay the foundations for more controllable, and naturally interactive conversational AI systems.