mlsanityData to 7 Oct 2026

Scaling 4D Representations

arXiv 2412.15212 ↗First posted 19 Dec 2024cs.CV cs.AI cs.LGcited by 30

Not on the map. Why?

Abstract (arXiv)

Scaling has not yet been convincingly demonstrated for pure self-supervised learning from video. However, prior work has focused evaluations on semantic-related tasks unicodex2013 action classification, ImageNet classification, etc. In this paper we focus on evaluating self-supervised learning on non-semantic vision tasks that are more spatial (3D) and temporal (+1D = 4D), such as camera pose estimation, point and object tracking, and depth estimation. We show that by learning from very large video datasets, masked auto-encoding (MAE) with transformer video models actually scales, consistently improving performance on these 4D tasks, as model size increases from 20M all the way to the largest by far reported self-supervised video model unicodex2013 22B parameters. Rigorous apples-to-apples comparison with many recent image and video models demonstrates the benefits of scaling 4D representations. Pretrained models are available at https://github.com/google-deepmind/representations4d .

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 66 of its 85 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

66 of 85 references.

References · 66

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  9. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2021Vision Transformers for Dense Predictioncited by 1,789
  12. 2021ViViT: A Video Vision Transformercited by 1,610
38 more
  1. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  2. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  3. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  4. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  6. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  7. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  9. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  10. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  11. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  12. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  13. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  14. 2017Look, Listen and Learncited by 452
  15. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  16. 2022Kubric: A scalable dataset generatorcited by 373
  17. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  18. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  19. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  20. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  21. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  22. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  23. 2017Objects that Soundcited by 254
  24. 2021Conditional Object-Centric Learning from Videocited by 247
  25. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  26. 2020Self-supervised Co-training for Video Representation Learningcited by 232
  27. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  28. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  29. 2020Atlas: End-to-End 3D Scene Reconstruction from Posed Imagescited by 206
  30. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  31. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  32. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  33. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 165
  34. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  35. 2016Learning Features by Watching Objects Movecited by 138
  36. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  37. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  38. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 10.

30 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  3. 2025Scaling Language-Free Visual Representation Learningcited by 62
  4. 2025AToken: A Unified Tokenizer for Visioncited by 30
  5. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 14
  6. 2025Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teacherscited by 11
  7. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  8. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 10
  9. 2026OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligencecited by 10
  10. 2025Recurrent Video Masked Autoencoderscited by 9
  11. 2025InternVideo-Next: Towards General Video Foundation Models without Video-Text Supervisioncited by 6
  12. 2025SemanticGen: Video Generation in Semantic Spacecited by 6
18 more
  1. 2025MoSiC: Optimal-Transport Motion Trajectory for Dense Self-Supervised Learningcited by 5
  2. 2025SciVid: Cross-Domain Evaluation of Video Models in Scientific Applicationscited by 5
  3. 2025Dynamic Reflections: Probing Video Representations with Text Alignmentcited by 5
  4. 2025VFMF: World Modeling by Forecasting Vision Foundation Model Featurescited by 5
  5. 2026You Don't Need Strong Assumptions: Visual Representation Learning via Temporal Differencescited by 5
  6. 2025Frozen Forecasting: A Unified Evaluationcited by 4
  7. 2025LayerLock: Non-collapsing Representation Learning with Progressive Freezingcited by 3
  8. 2025Probing and Leveraging Video Diffusion Transformer Features for Robust Point Trackingcited by 3
  9. 2025Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimationcited by 2
  10. 2025Unique Lives, Shared World: Learning from Single-Life Videoscited by 2
  11. 2026Gen4U: Unifying Video Generation and Understanding via Diffusioncited by 1
  12. 2026SeeSE3: Emergence of 3D Space in Vision Featurescited by 1
  13. 2025Tracking by Predicting 3-D Gaussians Over Timecited by 0
  14. 2026LA-Pose: Latent Action Pretraining Meets Pose Estimationcited by 0
  15. 2026Towards Data-Efficient Video Pre-training with Frozen Image Foundation Modelscited by 0
  16. 2026Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?cited by 0
  17. 2026Self-Supervised Learning of Structured Dynamics from Videoscited by 0
  18. 2026A Controlled Study of Self-Supervised Image and Video Pretraining under Limited Resourcescited by 0