mlsanityData to 7 Oct 2026

Attention Bottlenecks for Multimodal Fusion

arXiv 2107.00135 ↗First posted 30 Jun 2021cs.CVcited by 361

Not on the map. Why?

Abstract (arXiv)

Humans perceive the world by concurrently processing and fusing high-dimensional inputs from multiple modalities such as vision and audio. Machine perception models, in stark contrast, are typically modality-specific and optimised for unimodal benchmarks, and hence late-stage fusion of final representations or predictions from each modality (`late-fusion') is still a dominant paradigm for multimodal video classification. Instead, we introduce a novel transformer based architecture that uses `fusion bottlenecks' for modality fusion at multiple layers. Compared to traditional pairwise self-attention, our model forces information between different modalities to pass through a small number of bottleneck latents, requiring the model to collate and condense the most relevant information in each modality and only share what is necessary. We find that such a strategy improves fusion performance, at the same time reducing computational cost. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple audio-visual classification benchmarks including Audioset, Epic-Kitchens and VGGSound. All code and models will be released.

Key facts. Cited by 361 papers in the map (data through 7 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

53 of 69 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
38 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2016Deep Networks with Stochastic Depthcited by 973
  3. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  4. 2020Quantifying Attention Flow in Transformerscited by 759
  5. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  6. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  8. 2021AST: Audio Spectrogram Transformercited by 634
  9. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  10. 2017Look, Listen and Learncited by 452
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  12. 2020Multi-modal Transformer for Video Retrievalcited by 395
  13. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  14. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  15. 2017Temporal Relational Reasoning in Videoscited by 324
  16. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  17. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  18. 2017Objects that Soundcited by 254
  19. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  20. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  21. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  22. 2021Localizing Visual Sounds the Hard Waycited by 169
  23. 2020TEA: Temporal Excitation and Aggregation for Action Recognitioncited by 165
  24. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  25. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  26. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
  27. 2020Multi-modal Dense Video Captioningcited by 113
  28. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  29. 2016Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classificationcited by 84
  30. 2019More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregationcited by 64
  31. 2021Decoupling the Role of Data, Attention, and Losses in Multimodal Transformerscited by 63
  32. 2019Learning Spatio-Temporal Representation with Local and Global Diffusioncited by 61
  33. 2021Slow-Fast Auditory Streams For Audio Recognitioncited by 56
  34. 2020Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Soundscited by 55
  35. 2020Look Before you Speak: Visually Contextualized Utterancescited by 53
  36. 2020Parameter Efficient Multimodal Transformers for Video Representation Learningcited by 52
  37. 2019AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architecturescited by 46
  38. 2019Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervisioncited by 27

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 65; 2023: 107; 2024: 78; 2025: 66; 2026: 34.

361 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2023CoTracker: It is Better to Track Togethercited by 516
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  5. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  6. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  7. 2022Masked Autoencoders that Listencited by 271
  8. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  9. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  10. 2022Multiview Transformers for Video Recognitioncited by 148
  11. 2022UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detectioncited by 148
  12. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
38 more
  1. 2022Multimodal Token Fusion for Vision Transformerscited by 132
  2. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  3. 20234M: Massively Multimodal Masked Modelingcited by 118
  4. 2022ReSTR: Convolution-free Referring Image Segmentation Using Transformerscited by 112
  5. 2023Diffusion Models for Image Restoration and Enhancement: A Comprehensive Surveycited by 108
  6. 2023Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Predictioncited by 92
  7. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  8. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  9. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  10. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  11. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  12. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  13. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  14. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 72
  15. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  16. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  17. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  18. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  19. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  20. 2022MAViL: Masked Audio-Video Learnerscited by 56
  21. 2024ReconBoost: Boosting Can Achieve Modality Reconcilementcited by 56
  22. 2024Language Repository for Long Video Understandingcited by 55
  23. 2022Dynamic Multimodal Fusioncited by 54
  24. 2022MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray imagescited by 53
  25. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  26. 2022Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Groundingcited by 52
  27. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  28. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  29. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  30. 2022Audiovisual Masked Autoencoderscited by 47
  31. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  32. 2023Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baselinecited by 45
  33. 2023Enhancing multimodal cooperation via sample-level modality valuationcited by 43
  34. 2024GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformercited by 42
  35. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  36. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  37. 2021Audio-Visual Synchronisation in the wildcited by 39
  38. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39

50 most cited of 361 All 361 (JSON)