# Online and Linear-Time Attention by Enforcing Monotonic Alignments

arXiv: https://arxiv.org/abs/1704.00784 · HTML: https://mlsanity.com/paper/1704.00784/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Apr 2017.

## Built on by (most cited first, 30 of 85)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [LLaMA-Omni: Seamless Speech Interaction with Large Language Models](https://mlsanity.com/paper/2409.06666/) (2024)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [Translating Images into Maps](https://mlsanity.com/paper/2110.00966/) (2021)
- [DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations](https://mlsanity.com/paper/2403.06951/) (2024)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [DreamTuner: Single Image is Enough for Subject-Driven Generation](https://mlsanity.com/paper/2312.13691/) (2023)
- [Information-Transport-based Policy for Simultaneous Translation](https://mlsanity.com/paper/2210.12357/) (2022)
- [VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment](https://mlsanity.com/paper/2406.07855/) (2024)
- [TransLLaMa: LLM-based Simultaneous Translation System](https://mlsanity.com/paper/2402.04636/) (2024)
- [Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy](https://mlsanity.com/paper/2109.05238/) (2021)
- [Attention as a Guide for Simultaneous Speech Translation](https://mlsanity.com/paper/2212.07850/) (2022)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [RSDiff: Remote Sensing Image Generation from Text Using Diffusion Model](https://mlsanity.com/paper/2309.02455/) (2023)
- [Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling](https://mlsanity.com/paper/2509.08753/) (2025)
- [Hidden Markov Transformer for Simultaneous Machine Translation](https://mlsanity.com/paper/2303.00257/) (2023)
- [StyleSSP: Sampling StartPoint Enhancement for Training-free Diffusion-based Method for Style Transfer](https://mlsanity.com/paper/2501.11319/) (2025)
- [End-to-End Simultaneous Speech Translation with Differentiable Segmentation](https://mlsanity.com/paper/2305.16093/) (2023)
- [Modeling Dual Read/Write Paths for Simultaneous Machine Translation](https://mlsanity.com/paper/2203.09163/) (2022)
- [Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks](https://mlsanity.com/paper/2305.03101/) (2023)
- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Gaussian Multi-head Attention for Simultaneous Machine Translation](https://mlsanity.com/paper/2203.09072/) (2022)
- [Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework](https://mlsanity.com/paper/2203.09053/) (2022)
- [Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time](https://mlsanity.com/paper/2406.09569/) (2024)
- [Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translation](https://mlsanity.com/paper/2204.09595/) (2022)

## Abstract (arXiv)

Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time. We validate our approach on sentence summarization, machine translation, and online speech recognition problems and achieve results competitive with existing sequence-to-sequence models.