mlsanityData to 7 Oct 2026

data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language

arXiv 2202.03555 ↗First posted 7 Feb 2022cs.LGcited by 718

Not on the map. Why?

Abstract (arXiv)

While the general idea of self-supervised learning is identical across modalities, the actual algorithms and objectives differ widely because they were developed with a single modality in mind. To get us closer to general self-supervised learning, we present data2vec, a framework that uses the same learning method for either speech, NLP or computer vision. The core idea is to predict latent representations of the full input data based on a masked view of the input in a self-distillation setup using a standard Transformer architecture. Instead of predicting modality-specific targets such as words, visual tokens or units of human speech which are local in nature, data2vec predicts contextualized latent representations that contain information from the entire input. Experiments on the major benchmarks of speech recognition, image classification, and natural language understanding demonstrate a new state of the art or competitive performance to predominant approaches.

Key facts. Cited by 718 papers in the map (data through 7 Oct 2026). Builds on 67 of its 88 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

67 of 88 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  14. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  15. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  16. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  17. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  18. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  19. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  20. 2016Deep Networks with Stochastic Depthcited by 973
  21. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  22. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  23. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  24. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  25. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  26. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  27. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  28. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  29. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  30. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  31. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  32. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  33. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  34. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
  35. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  36. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  37. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  38. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 154; 2023: 197; 2024: 138; 2025: 107; 2026: 122.

718 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  3. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  4. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  5. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  6. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  7. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  8. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  9. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  10. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  11. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  12. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
38 more
  1. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  2. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  3. 2022Masked Autoencoders that Listencited by 271
  4. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  7. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  8. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  9. 2023Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion modelscited by 199
  10. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 196
  11. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  12. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  13. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  14. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  15. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 175
  16. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  17. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  18. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  19. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  20. 2022Learning 3D Representations from 2D Pre-trained Models via Image-to-Point Masked Autoencoderscited by 143
  21. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 124
  22. 2022Backbone is All Your Need: A Simplified Architecture for Visual Object Trackingcited by 119
  23. 20234M: Massively Multimodal Masked Modelingcited by 118
  24. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  25. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  26. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  27. 2022ConvMAE: Masked Convolution Meets Masked Autoencoderscited by 105
  28. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  29. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  30. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  31. 2022Language Models are General-Purpose Interfacescited by 96
  32. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  33. 2023Large-scale Training of Foundation Models for Wearable Biosignalscited by 89
  34. 2022SSL4EO-S12: A Large-Scale Multi-Modal, Multi-Temporal Dataset for Self-Supervised Learning in Earth Observationcited by 82
  35. 2022Object discovery and representation networkscited by 80
  36. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  37. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  38. 2023Decoupled Multimodal Distilling for Emotion Recognitioncited by 76

50 most cited of 718 All 718 (JSON)