mlsanityData to 14 Oct 2026

Generative Flows with Invertible Attentions

arXiv 2106.03959 ↗First posted 7 Jun 2021cs.LG cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

Flow-based generative models have shown an excellent ability to explicitly learn the probability density function of data via a sequence of invertible transformations. Yet, learning attentions in generative flows remains understudied, while it has made breakthroughs in other domains. To fill the gap, this paper introduces two types of invertible attention mechanisms, i.e., map-based and transformer-based attentions, for both unconditional and conditional generative flows. The key idea is to exploit a masked scheme of these two attentions to learn long-range data dependencies in the context of generative flows. The masked scheme allows for invertible attention modules with tractable Jacobian determinants, enabling its seamless integration at any positions of the flow-based models. The proposed attention mechanisms lead to more efficient generative flows, due to their capability of modeling the long-term data dependencies. Evaluation on multiple image synthesis tasks shows that the proposed attention flows result in efficient models and compare favorably against the state-of-the-art unconditional and conditional generative flows.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 54 of its 64 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

54 of 64 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2013Auto-Encoding Variational Bayescited by 4,053
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  10. 2016Improved Techniques for Training GANscited by 2,970
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  12. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
38 more
  1. 2017Non-local Neural Networkscited by 2,293
  2. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  3. 2016Density estimation using Real NVPcited by 2,042
  4. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,908
  5. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  6. 2018Demystifying MMD GANscited by 1,293
  7. 2018ESRGAN: Enhanced Super-Resolution Generative Adversarial Networkscited by 1,254
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  9. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  10. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  11. 2016Pixel Recurrent Neural Networkscited by 816
  12. 2018Self-Attention Generative Adversarial Networkscited by 809
  13. 2017Masked Autoregressive Flow for Density Estimationcited by 684
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  16. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  17. 2017Wasserstein Auto-Encoderscited by 373
  18. 2015Adversarial Autoencoderscited by 364
  19. 2019Attention Augmented Convolutional Networkscited by 363
  20. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 262
  21. 2019Residual Flows for Invertible Generative Modelingcited by 257
  22. 2020SRFlow: Learning the Super-Resolution Space with Normalizing Flowcited by 209
  23. 2018BAM: Bottleneck Attention Modulecited by 172
  24. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  25. 2018GANimation: Anatomically-aware Facial Animation from a Single Imagecited by 161
  26. 2021Generative Adversarial Transformerscited by 120
  27. 2017Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesiscited by 75
  28. 2019Sliced Wasserstein Generative Modelscited by 61
  29. 2019C-Flow: Conditional Generative Flow Models for Images and 3D Point Cloudscited by 50
  30. 2019Structured Output Learning with Conditional Generative Flowscited by 37
  31. 2018Sequential Attention GAN for Interactive Image Editingcited by 33
  32. 2018Attention-GAN for Object Transfiguration in Wild Imagescited by 32
  33. 2019Emerging Convolutions for Generative Normalizing Flowscited by 31
  34. 2020Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Searchcited by 25
  35. 2020AttentionNAS: Spatiotemporal Attention Cell Search for Video Classificationcited by 23
  36. 2017Wasserstein Divergence for GANscited by 22
  37. 2018DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)cited by 21
  38. 2019DUAL-GLOW: Conditional Flow-Based Generative Model for Modality Transfercited by 19

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 5; 2024: 2; 2025: 2; 2026: 2.

13 papers.

Sort:

  1. 2022On the Robustness of Normalizing Flows for Inverse Problems in Imagingcited by 8
  2. 2023Concurrent Misclassification and Out-of-Distribution Detection for Semantic Segmentation via Energy-Based Normalizing Flowcited by 8
  3. 2024Task-Driven Uncertainty Quantification in Inverse Problems via Conformal Predictioncited by 8
  4. 2025MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learningcited by 3
  5. 2023Neural Diffeomorphic Non-uniform B-spline Flowscited by 2
  6. 2025Conformal Bounds on Full-Reference Image Quality for Imaging Inverse Problemscited by 2
  7. 2023Hierarchical Disentangled Representation for Invertible Image Denoising and Beyondcited by 1
  8. 2024Probabilistic Forecasting of Irregular Time Series via Conditional Flowscited by 1
  9. 2022How Image Generation Helps Visible-to-Infrared Person Re-Identification?cited by 0
  10. 2023MixerFlow: MLP-Mixer meets Normalising Flowscited by 0
  11. 2023Dimension Mixer: Group Mixing of Input Dimensions for Efficient Function Approximationcited by 0
  12. 2026Generalizing Dynamics Modeling More Easily from Representation Perspectivecited by 0
1 more
  1. 2026α-Graph: Attention-Infused Normalizing Flow Approach to Tractable Graph Modelingcited by 0