mlsanityData to 7 Oct 2026

Scaling Language-Free Visual Representation Learning

arXiv 2504.01017 ↗First posted 1 Apr 2025cs.CVcited by 62

Not on the map. Why?

Abstract (arXiv)

Visual Self-Supervised Learning (SSL) currently underperforms Contrastive Language-Image Pretraining (CLIP) in multimodal settings such as Visual Question Answering (VQA). This multimodal gap is often attributed to the semantics introduced by language supervision, even though visual SSL and CLIP models are often trained on different data. In this work, we ask the question: "Do visual self-supervised approaches lag behind CLIP due to the lack of language supervision, or differences in the training data?" We study this question by training both visual SSL and CLIP models on the same MetaCLIP data, and leveraging VQA as a diverse testbed for vision encoders. In this controlled setup, visual SSL models scale better than CLIP models in terms of data and model capacity, and visual SSL performance does not saturate even after scaling up to 7B parameters. Consequently, we observe visual SSL methods achieve CLIP-level performance on a wide range of VQA and classic vision benchmarks. These findings demonstrate that pure visual SSL can match language-supervised visual pretraining at scale, opening new opportunities for vision-centric representation learning.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 91 of its 112 references that are in the map. First posted on arXiv 1 Apr 2025.

Builds on

91 of 112 references.

References · 91

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2017Mask R-CNNcited by 7,259
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022A ConvNet for the 2020scited by 3,545
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  13. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  15. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  16. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  17. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  18. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  19. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  20. 2019Towards VQA Models That Can Readcited by 1,696
  21. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  22. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  23. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  24. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  25. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  26. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  27. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  28. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  29. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  30. 2016Colorful Image Colorizationcited by 1,113
  31. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  32. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  33. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  34. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  35. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  36. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  37. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 806
  38. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 29.

62 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  4. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  5. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  6. 2025What matters for Representation Alignment: Global Information or Spatial Structure?cited by 76
  7. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  8. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 59
  9. 2025No Other Representation Component Is Needed: Diffusion Transformers Can Provide Representation Guidance by Themselvescited by 50
  10. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  11. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 34
  12. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
38 more
  1. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  2. 2026Improved Baselines with Representation Autoencoderscited by 24
  3. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  4. 2025Franca: Nested Matryoshka Clustering for Scalable Visual Representation Learningcited by 21
  5. 2026Reconstruction or Semantics? What Makes a Latent Space Useful for Robotic World Modelscited by 21
  6. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  7. 2025Data or Language Supervision: What Makes CLIP Better than DINO?cited by 13
  8. 2025In Pursuit of Pixel Supervision for Visual Pre-trainingcited by 12
  9. 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignmentcited by 12
  10. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 10
  11. 2026Representation Distribution Matching for One-Step Visual Generationcited by 10
  12. 2025Scaling Laws for Robust Comparison of Open Foundation Language-Vision Models and Datasetscited by 8
  13. 2026What Makes Video World Model Latents Action-Relevant: Prediction over Reconstructioncited by 8
  14. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Modelscited by 7
  15. 2025Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registrationcited by 7
  16. 2025Dynamic Reflections: Probing Video Representations with Text Alignmentcited by 5
  17. 2025SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinementcited by 4
  18. 2025Data Scaling Laws for Radiology Foundation Modelscited by 4
  19. 2026Nano World Models: A Minimalist Implementation of Future Video Predictioncited by 4
  20. 2025Learning Shared Representations from Unpaired Datacited by 3
  21. 2025RL makes MLLMs see better than SFTcited by 3
  22. 2025Midway Network: Learning Representations for Recognition and Motion from Latent Dynamicscited by 2
  23. 2025Data-Centric Lessons To Improve Speech-Language Pretrainingcited by 2
  24. 2025From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Imagescited by 2
  25. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  26. 2025DAVE: A VLM Vision Encoder for Document Understanding and Web Agentscited by 2
  27. 2026Learning Robust Visual Features in Computed Tomography Enables Efficient Transfer Learning for Clinical Taskscited by 2
  28. 2026Text-Conditional JEPA for Learning Semantically Rich Visual Representationscited by 2
  29. 2026Amortized Moment Matching for Visual Generationcited by 2
  30. 2025Towards Cellular-Scale Interpretability in Pathology Foundation Models for Biomarker Assessmentcited by 1
  31. 2025Is Generation Required for Data-Efficient Perception?cited by 1
  32. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimizationcited by 1
  33. 2026AudioMosaic: Contrastive Masked Audio Representation Learningcited by 1
  34. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  35. 2025Asymmetric Dual Self-Distillation for 3D Self-Supervised Representation Learningcited by 0
  36. 2025Why Prototypes Collapse: Diagnosing and Preventing Partial Collapse in Prototypical Self-Supervised Learningcited by 0
  37. 2026Decoupling Vision and Language: Codebook Anchored Visual Adaptationcited by 0
  38. 2026VINO: Video-driven Invariance for Non-contextual Objects via Structural Prior Guided De-contextualizationcited by 0

50 most cited of 62 All 62 (JSON)