mlsanityData to 14 Oct 2026

Visual Pre-Training on Unlabeled Images using Reinforcement Learning

arXiv 2506.11967 ↗First posted 13 Jun 2025cs.LG cs.CVcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

In reinforcement learning (RL), value-based algorithms learn to associate each observation with the states and rewards that are likely to be reached from it. We observe that many self-supervised image pre-training methods bear similarity to this formulation: learning features that associate crops of images with those of nearby views, e.g., by taking a different crop or color augmentation. In this paper, we complete this analogy and explore a method that directly casts pre-training on unlabeled image data like web crawls and video frames as an RL problem. We train a general value function in a dynamical system where an agent transforms an image by changing the view or adding image augmentations. Learning in this way resembles crop-consistency self-supervision, but through the reward function, offers a simple lever to shape feature learning using curated images or weakly labeled captions when they exist. Our experiments demonstrate improved representations when training on unlabeled images in the wild, including video data like EpicKitchens, scene data like COCO, and web-crawl data like CC12M.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 55 of its 67 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

55 of 67 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  7. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  12. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  13. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  14. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  15. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  16. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  17. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  18. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  19. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  20. 2020Meta Pseudo Labelscited by 374
  21. 2019S4L: Self-Supervised Semi-Supervised Learningcited by 335
  22. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  23. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  24. 2021A Survey on Deep Semi-supervised Learningcited by 251
  25. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  26. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  27. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  28. 2020Intriguing Properties of Contrastive Lossescited by 130
  29. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  30. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  31. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  32. 2023SILC: Improving Vision Language Pretraining with Self-Distillationcited by 66
  33. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  34. 2022A Theoretical Study of Inductive Biases in Contrastive Learningcited by 40
  35. 2023Is ImageNet worth 1 video? Learning strong image encoders from 1 long unlabelled videocited by 40
  36. 2017Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Taskscited by 38
  37. 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
  38. 2024Rethinking Patch Dependence for Masked Autoencoderscited by 29

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Reinforcement Learning with Action-Free Pre-Training from Videos
  2. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
  3. 2026From Pixels to Temporal Correlations: Learning Informative Representations for Reinforcement Learning Pre-training
  4. 2025Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning
  5. 2025Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
  6. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
  7. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning
  8. 2026UniDoc-RL: Coarse-to-Fine Visual RAG with Hierarchical Actions and Dense Rewards