# Variational Attention for Sequence-to-Sequence Models

arXiv: https://arxiv.org/abs/1712.08207 · HTML: https://mlsanity.com/paper/1712.08207/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2017.

## Built on by (most cited first, 29 of 29)

- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation](https://mlsanity.com/paper/2204.13031/) (2022)
- [Neural Processes with Stochastic Attention: Paying more attention to the context dataset](https://mlsanity.com/paper/2204.05449/) (2022)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Pathologies in priors and inference for Bayesian transformers](https://mlsanity.com/paper/2110.04020/) (2021)
- [Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors](https://mlsanity.com/paper/2204.01227/) (2022)
- [Probabilistic Attention for Interactive Segmentation](https://mlsanity.com/paper/2106.15338/) (2021)
- [An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation](https://mlsanity.com/paper/2209.14627/) (2022)
- [miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings](https://mlsanity.com/paper/2211.04928/) (2022)
- [Information Bottleneck Approach to Spatial Attention Learning](https://mlsanity.com/paper/2108.03418/) (2021)
- [PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism](https://mlsanity.com/paper/2212.09086/) (2022)
- [Hippocampal formation-inspired probabilistic generative model](https://mlsanity.com/paper/2103.07356/) (2021)
- [A study of latent monotonic attention variants](https://mlsanity.com/paper/2103.16710/) (2021)
- [MA-VAE: Multi-head Attention-based Variational Autoencoder Approach for Anomaly Detection in Multivariate Time-series Applied to Automotive Endurance Powertrain Testing](https://mlsanity.com/paper/2309.02253/) (2023)
- [Quinductor: a multilingual data-driven method for generating reading-comprehension questions using Universal Dependencies](https://mlsanity.com/paper/2103.10121/) (2021)
- [Treatment Learning Causal Transformer for Noisy Image Classification](https://mlsanity.com/paper/2203.15529/) (2022)
- [STREAM-VAE: Dual-Path Routing for Slow and Fast Dynamics in Vehicle Telemetry Anomaly Detection](https://mlsanity.com/paper/2511.15339/) (2025)
- [Discrete Auto-regressive Variational Attention Models for Text Modeling](https://mlsanity.com/paper/2106.08571/) (2021)
- [A Comprehensive Study of Vision Transformers in Image Classification Tasks](https://mlsanity.com/paper/2312.01232/) (2023)
- [TeVAE: A Variational Autoencoder Approach for Discrete Online Anomaly Detection in Variable-state Multivariate Time-series Data](https://mlsanity.com/paper/2407.06849/) (2024)
- [Smart Meter Data Anomaly Detection using Variational Recurrent Autoencoders with Attention](https://mlsanity.com/paper/2206.07519/) (2022)
- [DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusion](https://mlsanity.com/paper/2305.11517/) (2023)
- [Conditional Human Sketch Synthesis with Explicit Abstraction Control](https://mlsanity.com/paper/2306.09274/) (2023)
- [Variational Connectionist Temporal Classification for Order-Preserving Sequence Modeling](https://mlsanity.com/paper/2309.11983/) (2023)
- [Interactive Attention AI to translate low light photos to captions for night scene understanding in women safety](https://mlsanity.com/paper/2201.00969/) (2022)
- [Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation](https://mlsanity.com/paper/2305.15025/) (2023)
- [Incorporating Group Prior into Variational Inference for Tail-User Behavior Modeling in CTR Prediction](https://mlsanity.com/paper/2410.15098/) (2024)
- [ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learning](https://mlsanity.com/paper/2503.07506/) (2025)
- [Neurocircuitry-Inspired Hierarchical Graph Causal Attention Networks for Explainable Depression Identification](https://mlsanity.com/paper/2511.17622/) (2025)

## Abstract (arXiv)

The variational encoder-decoder (VED) encodes source information as a set of random variables using a neural network, which in turn is decoded into target data using another neural network. In natural language processing, sequence-to-sequence (Seq2Seq) models typically serve as encoder-decoder networks. When combined with a traditional (deterministic) attention mechanism, the variational latent space may be bypassed by the attention model, and thus becomes ineffective. In this paper, we propose a variational attention mechanism for VED, where the attention vector is also modeled as Gaussian distributed random variables. Results on two experiments show that, without loss of quality, our proposed method alleviates the bypassing phenomenon as it increases the diversity of generated sentences.