mlsanityData to 7 Oct 2026

Invertible Attention

arXiv 2106.09003 ↗First posted 16 Jun 2021cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Attention has been proved to be an efficient mechanism to capture long-range dependencies. However, so far it has not been deployed in invertible networks. This is due to the fact that in order to make a network invertible, every component within the network needs to be a bijective transformation, but a normal attention block is not. In this paper, we propose invertible attention that can be plugged into existing invertible models. We mathematically and experimentally prove that the invertibility of an attention model can be achieved by carefully constraining its Lipschitz constant. We validate the invertibility of our invertible attention on image reconstruction task with 3 popular datasets: CIFAR-10, SVHN, and CelebA. We also show that our invertible attention achieves similar performance in comparison with normal non-invertible attention on dense prediction tasks. The code is available at https://github.com/Schwartz-Zha/InvertibleAttention

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

32 of 52 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2017Graph Attention Networkscited by 8,854
  6. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2017Non-local Neural Networkscited by 2,290
  10. 2015Variational Inference with Normalizing Flowscited by 2,161
  11. 2016Density estimation using Real NVPcited by 2,029
  12. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
20 more
  1. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  2. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  3. 2018Image Super-Resolution Using Very Deep Residual Channel Attention Networkscited by 1,256
  4. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  5. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  6. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 913
  7. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  8. 2017Structure-measure: A New Way to Evaluate Foreground Mapscited by 480
  9. 2017Residual Attention Network for Image Classificationcited by 385
  10. 2018Invertible Residual Networkscited by 364
  11. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  12. 2021Anabranch Network for Camouflaged Object Segmentationcited by 227
  13. 2020SRFlow: Learning the Super-Resolution Space with Normalizing Flowcited by 209
  14. 2021Simultaneously Localize, Segment and Rank the Camouflaged Objectscited by 198
  15. 2020The Lipschitz Constant of Self-Attentioncited by 171
  16. 2019Pyramid Feature Attention Network for Saliency detectioncited by 128
  17. 2020Guiding Monocular Depth Estimation Using Depth-Attention Volumecited by 82
  18. 2020Parallax Attention for Unsupervised Stereo Correspondence Learningcited by 39
  19. 2018Memory Attention Networks for Skeleton-based Action Recognitioncited by 36
  20. 2019Emerging Convolutions for Generative Normalizing Flowscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024Masked Mixers for Language Generation and Retrievalcited by 2
  2. 2024Probabilistic Forecasting of Irregular Time Series via Conditional Flowscited by 1
  3. 2025Bringing together invertible UNets with invertible attention modules for memory-efficient diffusion modelscited by 0
  4. 2026Reversible Residual Normalization Alleviates Spatio-Temporal Distribution Shiftcited by 0
  5. 2026Riemannian Attention Mechanisms for Transformers: A Theoretical Framework and Architecture Designcited by 0