# Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU

arXiv: https://arxiv.org/abs/2109.07364 · HTML: https://mlsanity.com/paper/2109.07364/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Random Feature Attention](https://mlsanity.com/paper/2103.02143/) (2021)
- [Finetuning Pretrained Transformers into RNNs](https://mlsanity.com/paper/2103.13076/) (2021)
- [A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1907.00390/) (2019)
- [Future-Guided Incremental Transformer for Simultaneous Translation](https://mlsanity.com/paper/2012.12465/) (2020)
- [Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU](https://mlsanity.com/paper/2010.05330/) (2020)

## Built on by (most cited first, 17 of 17)

- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials](https://mlsanity.com/paper/2205.05467/) (2022)
- [Mitigating Over-smoothing in Transformers via Regularized Nonlocal Functionals](https://mlsanity.com/paper/2312.00751/) (2023)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model](https://mlsanity.com/paper/2305.10845/) (2023)
- [Efficient Encoders for Streaming Sequence Tagging](https://mlsanity.com/paper/2301.09244/) (2023)
- [Real-time Caller Intent Detection In Human-Human Customer Support Spoken Conversations](https://mlsanity.com/paper/2208.06802/) (2022)
- [Sentence-Incremental Neural Coreference Resolution](https://mlsanity.com/paper/2305.16947/) (2023)
- [The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling](https://mlsanity.com/paper/2307.15508/) (2023)
- [Enhanced Simultaneous Machine Translation with Word-level Policies](https://mlsanity.com/paper/2310.16417/) (2023)
- [Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models](https://mlsanity.com/paper/2501.00953/) (2025)
- [A Major Obstacle for NLP Research: Let's Talk about Time Allocation!](https://mlsanity.com/paper/2211.16858/) (2022)
- [Learning to generate and corr- uh I mean repair language in real-time](https://mlsanity.com/paper/2308.11683/) (2023)
- [When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality](https://mlsanity.com/paper/2402.13113/) (2024)
- [Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing](https://mlsanity.com/paper/2310.18229/) (2023)
- [Dialogue with Robots: Proposals for Broadening Participation and Research in the SLIVAR Community](https://mlsanity.com/paper/2404.01158/) (2024)
- [GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?](https://mlsanity.com/paper/2610.15129/) (2026)

## Abstract (arXiv)

Incremental processing allows interactive systems to respond based on partial inputs, which is a desirable property e.g. in dialogue agents. The currently popular Transformer architecture inherently processes sequences as a whole, abstracting away the notion of time. Recent work attempts to apply Transformers incrementally via restart-incrementality by repeatedly feeding, to an unchanged model, increasingly longer input prefixes to produce partial outputs. However, this approach is computationally costly and does not scale efficiently for long sequences. In parallel, we witness efforts to make Transformers more efficient, e.g. the Linear Transformer (LT) with a recurrence mechanism. In this work, we examine the feasibility of LT for incremental NLU in English. Our results show that the recurrent LT model has better incremental performance and faster inference speed compared to the standard Transformer and LT with restart-incrementality, at the cost of part of the non-incremental (full sequence) quality. We show that the performance drop can be mitigated by training the model to wait for right context before committing to an output and that training with input prefixes is beneficial for delivering correct partial outputs.