mlsanityData to 7 Oct 2026

TAPIR: Learning Adaptive Revision for Incremental Natural Language Understanding with a Two-Pass Model

arXiv 2305.10845 ↗First posted 18 May 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Language is by its very nature incremental in how it is produced and processed. This property can be exploited by NLP systems to produce fast responses, which has been shown to be beneficial for real-time interactive applications. Recent neural network-based approaches for incremental processing mainly use RNNs or Transformers. RNNs are fast but monotonic (cannot correct earlier output, which can be necessary in incremental processing). Transformers, on the other hand, consume whole sequences, and hence are by nature non-incremental. A restart-incremental interface that repeatedly passes longer input prefixes can be used to obtain partial outputs, while providing the ability to revise. However, this method becomes costly as the sentence grows longer. In this work, we propose the Two-pass model for AdaPtIve Revision (TAPIR) and introduce a method to obtain an incremental supervision signal for learning an adaptive revision policy. Experimental results on sequence labelling show that our model has better incremental performance and faster inference speed compared to restart-incremental Transformers, while showing little degradation on full sequences.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 38 of its 72 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

38 of 72 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  6. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  7. 2014Neural Turing Machinescited by 630
  8. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  9. 2014Memory Networkscited by 354
  10. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  11. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  12. 2018Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialogcited by 124
26 more
  1. 2019Monotonic Infinite Lookback Attention for Simultaneous Machine Translationcited by 103
  2. 2016Learning to Translate in Real-time with Neural Machine Translationcited by 97
  3. 2019Monotonic Multihead Attentioncited by 91
  4. 2017Online and Linear-Time Attention by Enforcing Monotonic Alignmentscited by 85
  5. 2019A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Fillingcited by 85
  6. 2017Monotonic Chunkwise Attentioncited by 82
  7. 2018Retrieve and Refine: Improved Sequence Generation Models For Dialoguecited by 82
  8. 2016Improving Neural Language Models with a Continuous Cachecited by 81
  9. 2019Two-Pass End-to-End Speech Recognitioncited by 62
  10. 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
  11. 2018The Importance of Being Recurrent for Modeling Hierarchical Structurecited by 42
  12. 2019Simpler and Faster Learning of Adaptive Policies for Simultaneous Translationcited by 36
  13. 2019Incremental Transformer with Deliberation Decoder for Document Grounded Conversationscited by 35
  14. 2020Re-translation versus Streaming for Simultaneous Translationcited by 30
  15. 2018Reversible Recurrent Neural Networkscited by 22
  16. 2021Towards Incremental Transformers: An Empirical Analysis of Transformer Models for Incremental NLUcited by 16
  17. 2022Turning Fixed to Adaptive: Integrating Post-Evaluation into Simultaneous Machine Translationcited by 14
  18. 2020Incremental Processing in the Age of Non-Incremental Encoders: An Empirical Assessment of Bidirectional Models for Incremental NLUcited by 13
  19. 2021Deliberation of Streaming RNN-Transducer by Non-autoregressive Decodingcited by 13
  20. 2022Teaching BERT to Wait: Balancing Accuracy and Latency for Streaming Disfluency Detectioncited by 12
  21. 2020Parallel Rescoring with Transformer for Streaming On-Device Speech Recognitioncited by 9
  22. 2018Incremental Natural Language Processing: Challenges, Strategies, and Evaluationcited by 8
  23. 2019The Referential Reader: A Recurrent Entity Network for Anaphora Resolutioncited by 8
  24. 2020Opportunistic Decoding with Timely Correction for Simultaneous Translationcited by 8
  25. 2022Implicit N-grams Induced by Recurrencecited by 4
  26. 2023Efficient Encoders for Streaming Sequence Taggingcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1; 2025: 1.

5 papers.

  1. 2023The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labellingcited by 2
  2. 2025Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Modelscited by 2
  3. 2023Learning to generate and corr- uh I mean repair language in real-timecited by 1
  4. 2024When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementalitycited by 1
  5. 2023Revising with a Backward Glance: Regressions and Skips during Reading as Cognitive Signals for Revision Policies in Incremental Processingcited by 0