mlsanityData to 7 Oct 2026

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

arXiv 2104.11178 ↗First posted 22 Apr 2021cs.CV cs.AI cs.LGcited by 427

Not on the map. Why?

Abstract (arXiv)

We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our Video-Audio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT's vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT's audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT's source code is publicly available.

Key facts. Cited by 427 papers in the map (data through 7 Oct 2026). Builds on 96 of its 118 references that are in the map. First posted on arXiv 22 Apr 2021.

Builds on

96 of 118 references.

References · 96

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2016Layer Normalizationcited by 5,579
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  9. 2018CBAM: Convolutional Block Attention Modulecited by 2,463
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2017Non-local Neural Networkscited by 2,290
  12. 2018SlowFast Networks for Video Recognitioncited by 1,862
  13. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  14. 2021ViViT: A Video Vision Transformercited by 1,610
  15. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  16. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  17. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  18. 2020Point Transformercited by 1,462
  19. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  20. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  21. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  22. 2019Contrastive Multiview Codingcited by 1,297
  23. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  24. 2016Colorful Image Colorizationcited by 1,113
  25. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  26. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  27. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  28. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  29. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  30. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  31. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  32. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  33. 2017Convolutional Sequence to Sequence Learningcited by 550
  34. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  35. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  36. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  37. 2017Look, Listen and Learncited by 452
  38. 2015Unsupervised Learning of Video Representations using LSTMscited by 434

50 most cited of 96 All 96 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 124; 2023: 109; 2024: 79; 2025: 46; 2026: 31.

427 papers.

Sort:

By area:

  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  4. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  5. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  8. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  9. 2022Multimodal Learning with Transformers: A Surveycited by 399
  10. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  11. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
38 more
  1. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  2. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 259
  5. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  6. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  7. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  8. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  9. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  10. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  11. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  12. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  13. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  14. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  15. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  16. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  17. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  18. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  19. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  20. 2022FlexiViT: One Model for All Patch Sizescited by 151
  21. 2022Multiview Transformers for Video Recognitioncited by 148
  22. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  23. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  24. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  25. 2022TransTab: Learning Transferable Tabular Transformers Across Tablescited by 132
  26. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  27. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  28. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  29. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  30. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  31. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  32. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  33. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  34. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  35. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  36. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  37. 2023On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligencecited by 88
  38. 2022Self-Supervised Learning for Videos: A Surveycited by 85

50 most cited of 427 All 427 (JSON)