mlsanityData to 7 Oct 2026

Variational Attention for Sequence-to-Sequence Models

arXiv 1712.08207 ↗First posted 21 Dec 2017cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

The variational encoder-decoder (VED) encodes source information as a set of random variables using a neural network, which in turn is decoded into target data using another neural network. In natural language processing, sequence-to-sequence (Seq2Seq) models typically serve as encoder-decoder networks. When combined with a traditional (deterministic) attention mechanism, the variational latent space may be bypassed by the attention model, and thus becomes ineffective. In this paper, we propose a variational attention mechanism for VED, where the attention vector is also modeled as Gaussian distributed random variables. Results on two experiments show that, without loss of quality, our proposed method alleviates the bypassing phenomenon as it increases the diversity of generated sentences.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 11; 2023: 6; 2024: 2; 2025: 3.

29 papers.

Sort:

By area:

  1. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  2. 2022DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generationcited by 35
  3. 2022Neural Processes with Stochastic Attention: Paying more attention to the context datasetcited by 16
  4. 2022Recent Advances in Neural Text Generation: A Task-Agnostic Surveycited by 14
  5. 2021Pathologies in priors and inference for Bayesian transformerscited by 13
  6. 2022Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priorscited by 13
  7. 2021Probabilistic Attention for Interactive Segmentationcited by 12
  8. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  9. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7
  10. 2021Information Bottleneck Approach to Spatial Attention Learningcited by 6
  11. 2022PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanismcited by 6
  12. 2021Hippocampal formation-inspired probabilistic generative modelcited by 5
17 more
  1. 2021A study of latent monotonic attention variantscited by 5
  2. 2023MA-VAE: Multi-head Attention-based Variational Autoencoder Approach for Anomaly Detection in Multivariate Time-series Applied to Automotive Endurance Powertrain Testingcited by 4
  3. 2021Quinductor: a multilingual data-driven method for generating reading-comprehension questions using Universal Dependenciescited by 3
  4. 2022Treatment Learning Causal Transformer for Noisy Image Classificationcited by 3
  5. 2021Discrete Auto-regressive Variational Attention Models for Text Modelingcited by 2
  6. 2023A Comprehensive Study of Vision Transformers in Image Classification Taskscited by 2
  7. 2024TeVAE: A Variational Autoencoder Approach for Discrete Online Anomaly Detection in Variable-state Multivariate Time-series Datacited by 2
  8. 2025STREAM-VAE: Dual-Path Routing for Slow and Fast Dynamics in Vehicle Telemetry Anomaly Detectioncited by 2
  9. 2022Smart Meter Data Anomaly Detection using Variational Recurrent Autoencoders with Attentioncited by 1
  10. 2023DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusioncited by 1
  11. 2023Conditional Human Sketch Synthesis with Explicit Abstraction Controlcited by 1
  12. 2023Variational Connectionist Temporal Classification for Order-Preserving Sequence Modelingcited by 1
  13. 2022Interactive Attention AI to translate low light photos to captions for night scene understanding in women safetycited by 0
  14. 2023Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generationcited by 0
  15. 2024Incorporating Group Prior into Variational Inference for Tail-User Behavior Modeling in CTR Predictioncited by 0
  16. 2025ADROIT: A Self-Supervised Framework for Learning Robust Representations for Active Learningcited by 0
  17. 2025Neurocircuitry-Inspired Hierarchical Graph Causal Attention Networks for Explainable Depression Identificationcited by 0