# The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling

arXiv: https://arxiv.org/abs/2307.15508 · HTML: https://mlsanity.com/paper/2307.15508/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 54 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [SimulEval: An Evaluation Toolkit for Simultaneous Translation](https://mlsanity.com/paper/2007.16193/) (2020)
- [Simultaneous Translation Policies: From Fixed to Adaptive](https://mlsanity.com/paper/2004.13169/) (2020)
- [Re-translation versus Streaming for Simultaneous Translation](https://mlsanity.com/paper/2004.03643/) (2020)
- [Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU](https://mlsanity.com/paper/2109.07364/) (2021)
- [Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU](https://mlsanity.com/paper/2010.05330/) (2020)
- [Annotation and Classification of Sentence-level Revision Improvement](https://mlsanity.com/paper/1909.05309/) (2019)
- [Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detection](https://mlsanity.com/paper/2205.00620/) (2022)
- [Incremental Natural Language Processing: Challenges, Strategies, and Evaluation](https://mlsanity.com/paper/1805.12518/) (2018)
- [Opportunistic Decoding with Timely Correction for Simultaneous Translation](https://mlsanity.com/paper/2005.00675/) (2020)
- [TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model](https://mlsanity.com/paper/2305.10845/) (2023)
- [Assessing incrementality in sequence-to-sequence models](https://mlsanity.com/paper/1906.03293/) (2019)
- [Efficient Encoders for Streaming Sequence Tagging](https://mlsanity.com/paper/2301.09244/) (2023)
- [Real-time Caller Intent Detection In Human-Human Customer Support Spoken Conversations](https://mlsanity.com/paper/2208.06802/) (2022)
- [Sentence-Incremental Neural Coreference Resolution](https://mlsanity.com/paper/2305.16947/) (2023)

## Built on by (most cited first, 2 of 2)

- [When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality](https://mlsanity.com/paper/2402.13113/) (2024)
- [Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing](https://mlsanity.com/paper/2310.18229/) (2023)

## Abstract (arXiv)

Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.