mlsanityData to 14 Oct 2026

Video-STLayout Pre-training

arXiv 2609.24031 ↗First posted 21 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In recent years, pre-training has become fundamental to learning effective video representations, enabling strong transfer to downstream tasks. A popular framework in pre-training involves aligning features of a video encoder with that of another modality, for example, language or audio. We introduce Video-STLayout pre-training, a novel strategy for obtaining rich video representations informed by spatio-temporal layout of object bounding boxes. Object layouts can easily be obtained by applying an off-the-shelf object detector on the video frames. Our method uses a contrastive loss to align video features with the layout features from a trained layout encoder. We show the effectiveness of our approach in the task of activity recognition in complex scenes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 57 of its 61 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

57 of 61 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2017Mask R-CNNcited by 7,277
  9. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  2. 2015Fast R-CNNcited by 2,933
  3. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  4. 2018SlowFast Networks for Video Recognitioncited by 1,867
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  7. 2021Video Swin Transformercited by 1,091
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2021Multiscale Vision Transformerscited by 897
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  13. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  14. 2017Learning multiple visual domains with residual adapterscited by 557
  15. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  16. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  17. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  18. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  19. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  20. 2017Temporal Segment Networks for Action Recognition in Videoscited by 279
  21. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  22. 2023Video Understanding with Large Language Models: A Surveycited by 206
  23. 2020SpeedNet: Learning the Speediness in Videoscited by 171
  24. 2021BEVT: BERT Pretraining of Video Transformerscited by 169
  25. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  26. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  27. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  28. 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
  29. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  30. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 114
  31. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  32. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  33. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  34. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  35. 2021Object-aware Video-language Pre-training for Retrievalcited by 57
  36. 2022Masked Motion Encoding for Self-Supervised Video Representation Learningcited by 39
  37. 2024OphCLIP: Hierarchical Retrieval-Augmented Learning for Ophthalmic Surgical Video-Language Pretrainingcited by 29
  38. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.