# Efficient Encoders for Streaming Sequence Tagging

arXiv: https://arxiv.org/abs/2301.09244 · HTML: https://mlsanity.com/paper/2301.09244/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 45 references that are in the map.
- First posted on arXiv 23 Jan 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Learning to Translate in Real-time with Neural Machine Translation](https://mlsanity.com/paper/1610.00388/) (2016)
- [Monotonic Multihead Attention](https://mlsanity.com/paper/1909.12406/) (2019)
- [Can neural machine translation do simultaneous translation?](https://mlsanity.com/paper/1606.02012/) (2016)
- [Online and Linear-Time Attention by Enforcing Monotonic Alignments](https://mlsanity.com/paper/1704.00784/) (2017)
- [Monotonic Chunkwise Attention](https://mlsanity.com/paper/1712.05382/) (2017)
- [Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation](https://mlsanity.com/paper/1806.03661/) (2018)
- [Strongly Incremental Constituency Parsing with Graph Neural Networks](https://mlsanity.com/paper/2010.14568/) (2020)
- [Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLU](https://mlsanity.com/paper/2109.07364/) (2021)
- [Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLU](https://mlsanity.com/paper/2010.05330/) (2020)
- [Differentiable Adaptive Computation Time for Visual Reasoning](https://mlsanity.com/paper/2004.12770/) (2020)
- [DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference](https://mlsanity.com/paper/2109.11745/) (2021)
- [From Simultaneous to Streaming Machine Translation by Leveraging Streaming History](https://mlsanity.com/paper/2203.02459/) (2022)
- [Incremental Natural Language Processing: Challenges, Strategies, and Evaluation](https://mlsanity.com/paper/1805.12518/) (2018)
- [Streaming Intended Query Detection using E2E Modeling for Continued Conversation](https://mlsanity.com/paper/2208.13322/) (2022)
- [CALM: Continuous Adaptive Learning for Language Modeling](https://mlsanity.com/paper/2004.03794/) (2020)

## Built on by (most cited first, 4 of 4)

- [TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model](https://mlsanity.com/paper/2305.10845/) (2023)
- [The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling](https://mlsanity.com/paper/2307.15508/) (2023)
- [When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality](https://mlsanity.com/paper/2402.13113/) (2024)
- [Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processing](https://mlsanity.com/paper/2310.18229/) (2023)

## Abstract (arXiv)

A naive application of state-of-the-art bidirectional encoders for streaming sequence tagging would require encoding each token from scratch for each new token in an incremental streaming input (like transcribed speech). The lack of re-usability of previous computation leads to a higher number of Floating Point Operations (or FLOPs) and higher number of unnecessary label flips. Increased FLOPs consequently lead to higher wall-clock time and increased label flipping leads to poorer streaming performance. In this work, we present a Hybrid Encoder with Adaptive Restart (HEAR) that addresses these issues while maintaining the performance of bidirectional encoders over the offline (or complete) inputs while improving performance on streaming (or incomplete) inputs. HEAR has a Hybrid unidirectional-bidirectional encoder architecture to perform sequence tagging, along with an Adaptive Restart Module (ARM) to selectively guide the restart of bidirectional portion of the encoder. Across four sequence tagging tasks, HEAR offers FLOP savings in streaming settings upto 71.1% and also outperforms bidirectional encoders for streaming predictions by upto +10% streaming exact match.