mlsanityData to 14 Oct 2026

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

arXiv 2609.33419 ↗First posted 27 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective, data exposure, schedule, scale, and decoder capacity can all vary at once. This makes it hard to identify which choices yield motion-prioritized representations, whose gains concentrate on frame-to-frame change while retaining useful appearance. We address this with a matched 4 × 6 = 24 architecture-objective study at roughly 170M ~ 190M encoder scale on ~1.7M OpenVid and Moments-in-Time v2 clips for 8 epochs, and propose TT-VidT. TT-VidT combines a DINOv3-initialized ViT-B/16 per-frame spatial path with a compact Temporal Transfer Layer, trained by Diff Compression to reconstruct target frames from a first-frame appearance anchor and frame-specific motion tokens. The sweep shows that TT3D with Diff Compression, not either component alone, enters the strongest motion-sensitive regime, and decoder ablations favor a compact video-pretrained decoder. In final comparison, TT-VidT leads Jester, Something-Something V2, ARID, and Diving48 fine-tuning simultaneously, improving over the strongest non-TT row by 54% ~ 121%, while using 48% fewer encoder FLOPs than DisMo and 55% fewer than VideoMAE or V-JEPA2. HMDB51, IARD, and EPIC-Kitchens bound the claim.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 52 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

40 of 52 references.

References · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  4. 2022Scalable Diffusion Models with Transformerscited by 6,054
  5. 2022Flow Matching for Generative Modelingcited by 5,702
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2021ViViT: A Video Vision Transformercited by 1,620
  8. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  9. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  10. 2016Deep Variational Information Bottleneckcited by 930
  11. 2020Rescaling Egocentric Visioncited by 540
  12. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
28 more
  1. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  2. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  3. 2018Moments in Time Dataset: one million videos for event understandingcited by 215
  4. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  5. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  6. 2016Information Dropout: Learning Optimal Representations Through Noisy Computationcited by 132
  7. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  8. 2020ARID: A New Dataset for Recognizing Action in the Darkcited by 43
  9. 2022AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoderscited by 43
  10. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  11. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 32
  12. 2022A Deeper Dive Into What Deep Spatiotemporal Networks Encode: Quantifying Static vs. Dynamic Informationcited by 26
  13. 2021No frame left behind: Full Video Action Recognitioncited by 24
  14. 2022How Severe is Benchmark-Sensitivity in Video Self-Supervised Learning?cited by 23
  15. 2025An Empirical Study of Autoregressive Pre-training from Videoscited by 21
  16. 2022Self-supervised Video Representation Learning with Motion-Aware Masked Autoencoderscited by 20
  17. 2023ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Videocited by 14
  18. 2025Rethinking JEPA: Compute-Efficient Video SSL with Frozen Teacherscited by 12
  19. 2023No More Shortcuts: Realizing the Potential of Temporal Self-Supervisioncited by 11
  20. 2025DisMo: Disentangled Motion Representations for Open-World Motion Transfercited by 10
  21. 2025SMILE: Infusing Spatial and Motion Semantics in Masked Video Learningcited by 9
  22. 2025ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognitioncited by 5
  23. 2025SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learningcited by 4
  24. 2025VideoMAP: Toward Scalable Mamba-based Video Autoregressive Pretrainingcited by 3
  25. 2026VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latentscited by 3
  26. 2025FRAME: Pre-Training Video Feature Representations via Anticipation and Memorycited by 2
  27. 2025Advancing Video Self-Supervised Learning via Image Foundation Modelscited by 1
  28. 2025Learning from Next-Frame Prediction: Autoregressive Video Modeling Encodes Effective Representationscited by 1

Built on by

Not cited yet.