mlsanityData to 7 Oct 2026

Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language

arXiv 2212.07525 ↗First posted 14 Dec 2022cs.LG cs.CL cs.SDcited by 108

Not on the map. Why?

Abstract (arXiv)

Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8% with a ViT-L model trained for 150 epochs.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 56 of its 67 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

56 of 67 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2016Layer Normalizationcited by 5,579
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  13. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  14. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  17. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  18. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  19. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  20. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  21. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  22. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  23. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  24. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  25. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
  26. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  27. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  28. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  29. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  30. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  31. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  32. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  33. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  34. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  35. 2021Unsupervised Speech Recognitioncited by 147
  36. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  37. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  38. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 24; 2024: 33; 2025: 32; 2026: 19.

108 papers.

Sort:

By area:

  1. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  2. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  3. 2023A Cookbook of Self-Supervised Learningcited by 248
  4. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  5. 2023CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoderscited by 167
  6. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  7. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  8. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  9. 2024EEG2Rep: Enhancing Self-supervised EEG Representation Through Informative Masked Inputscited by 48
  10. 2024EmoBox: Multilingual Multi-corpus Speech Emotion Recognition Toolkit and Benchmarkcited by 39
  11. 2023Transformers in Speech Processing: A Surveycited by 38
  12. 2024MedCLIP-SAM: Bridging Text and Image Towards Universal Medical Image Segmentationcited by 37
38 more
  1. 2024MedCLIP-SAMv2: Towards Universal Text-Driven Medical Image Segmentationcited by 37
  2. 2026Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generationcited by 33
  3. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
  4. 2024SyllableLM: Learning Coarse Semantic Units for Speech Language Modelscited by 24
  5. 2023How to Scale Your EMAcited by 22
  6. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  7. 2024MIM-Refiner: A Contrastive Learning Boost from Intermediate Pre-Trained Representationscited by 19
  8. 2025CheXWorld: Exploring Image World Modeling for Radiograph Representation Learningcited by 18
  9. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  10. 2024Learning General Representation of 12-Lead Electrocardiogram with a Joint-Embedding Predictive Architecturecited by 17
  11. 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
  12. 2024One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representationcited by 16
  13. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  14. 2024CR-CTC: Consistency regularization on CTC for improved speech recognitioncited by 15
  15. 2024animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacousticscited by 14
  16. 2023A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Taskscited by 13
  17. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  18. 2024S-JEPA: towards seamless cross-dataset transfer through dynamic spatial attentioncited by 13
  19. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  20. 2025EchoWorld: Learning Motion-Aware World Models for Echocardiography Probe Guidancecited by 13
  21. 2023Protein-ligand binding representation learning from fine-grained interactionscited by 12
  22. 2023Representation Deficiency in Masked Language Modelingcited by 10
  23. 2023Fast-HuBERT: An Efficient Training Framework for Self-Supervised Speech Representation Learningcited by 9
  24. 2024Open Implementation and Study of BEST-RQ for Speech Processingcited by 9
  25. 2025Dolphin: A Large-Scale Automatic Speech Recognition Model for Eastern Languagescited by 8
  26. 2023Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separatorcited by 7
  27. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  28. 2024SER Evals: In-domain and Out-of-domain Benchmarking for Speech Emotion Recognitioncited by 6
  29. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  30. 2026Earth System Foundation Model (ESFM): A unified framework for heterogeneous data integration and forecastingcited by 6
  31. 2023Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encodercited by 5
  32. 2024Revisiting ASR Error Correction with Specialized Modelscited by 5
  33. 2024Sample- and Parameter-Efficient Auto-Regressive Image Modelscited by 5
  34. 2025Towards Early Prediction of Self-Supervised Speech Model Performancecited by 5
  35. 2025Steering Language Model to Stable Speech Emotion Recognition via Contextual Perception and Chain of Thoughtcited by 5
  36. 2025UniWav: Towards Unified Pre-training for Speech Representation Learning and Generationcited by 5
  37. 2025Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representationcited by 5
  38. 2025Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Predictioncited by 5

50 most cited of 108 All 108 (JSON)