# Assessing incrementality in sequence-to-sequence models

arXiv: https://arxiv.org/abs/1906.03293 · HTML: https://mlsanity.com/paper/1906.03293/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2019.

## Built on by (most cited first, 4 of 4)

- [From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency](https://mlsanity.com/paper/2404.12145/) (2024)
- [Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models](https://mlsanity.com/paper/2412.05353/) (2024)
- [The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling](https://mlsanity.com/paper/2307.15508/) (2023)
- [When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality](https://mlsanity.com/paper/2402.13113/) (2024)

## Abstract (arXiv)

Since their inception, encoder-decoder models have successfully been applied to a wide array of problems in computational linguistics. The most recent successes are predominantly due to the use of different variations of attention mechanisms, but their cognitive plausibility is questionable. In particular, because past representations can be revisited at any point in time, attention-centric methods seem to lack an incentive to build up incrementally more informative representations of incoming sentences. This way of processing stands in stark contrast with the way in which humans are believed to process language: continuously and rapidly integrating new information as it is encountered. In this work, we propose three novel metrics to assess the behavior of RNNs with and without an attention mechanism and identify key differences in the way the different model types process sentences.