mlsanityData to 14 Oct 2026

Dual-path Adaptation from Image to Video Transformers

arXiv 2303.09857 ↗First posted 17 Mar 2023cs.CV cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

In this paper, we efficiently transfer the surpassing representation power of the vision foundation models, such as ViT and Swin, for video understanding with only a few trainable parameters. Previous adaptation methods have simultaneously considered spatial and temporal modeling with a unified learnable module but still suffered from fully leveraging the representative capabilities of image transformers. We argue that the popular dual-path (two-stream) architecture in video models can mitigate this problem. We propose a novel DualPath adaptation separated into spatial and temporal adaptation paths, where a lightweight bottleneck adapter is employed in each transformer block. Especially for temporal dynamic modeling, we incorporate consecutive frames into a grid-like frameset to precisely imitate vision transformers' capability that extrapolates relationships between tokens. In addition, we extensively investigate the multiple baselines from a unified perspective in video understanding and compare them with DualPath. Experimental results on four action recognition benchmarks prove that pretrained image transformers with DualPath can be effectively generalized beyond the data domain.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 77 of its 86 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

77 of 86 references.

References · 77

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2016Layer Normalizationcited by 5,669
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  9. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  11. 2017The Kinetics Human Action Video Datasetcited by 1,909
  12. 2018SlowFast Networks for Video Recognitioncited by 1,867
  13. 2022Visual Prompt Tuningcited by 1,681
  14. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  15. 2021ViViT: A Video Vision Transformercited by 1,620
  16. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  17. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  18. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  19. 2017Random Erasing Data Augmentationcited by 1,239
  20. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  21. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  22. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
  23. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  24. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  25. 2021Video Swin Transformercited by 1,091
  26. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  27. 2021Scaling Vision Transformerscited by 999
  28. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  29. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  30. 2021Multiscale Vision Transformerscited by 897
  31. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  32. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  33. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  34. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  35. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  36. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  37. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 527
  38. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 14; 2026: 10.

41 papers.

Sort:

By area:

  1. 2024CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognitioncited by 61
  2. 2024Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognitioncited by 49
  3. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  4. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 36
  5. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 26
  6. 2023STA-VPT: SpatioTemporally Aligned Visual Prompt Tuningcited by 17
  7. 2024M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognitioncited by 11
  8. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 11
  9. 2024C2C: Component-to-Composition Learning for Zero-Shot Compositional Action Recognitioncited by 9
  10. 2025SMILE: Infusing Spatial and Motion Semantics in Masked Video Learningcited by 9
  11. 2023Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognitioncited by 8
  12. 2024Rethinking Image-to-Video Adaptation: An Object-centric Perspectivecited by 8
29 more
  1. 2025SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Costcited by 8
  2. 2023MoSA: Mixture of Sparse Adapters for Visual Efficient Tuningcited by 6
  3. 2024SSFam: Scribble Supervised Salient Object Detection Familycited by 6
  4. 2025Bootstrap Your Own Views: Masked Ego-Exo Modeling for Fine-grained View-invariant Video Representationscited by 6
  5. 2023MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognitioncited by 4
  6. 2025Faster Parameter-Efficient Tuning with Token Redundancy Reductioncited by 4
  7. 2025MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognitioncited by 4
  8. 2024SurgPETL: Parameter-Efficient Image-to-Surgical-Video Transfer Learning for Surgical Phase Recognitioncited by 3
  9. 2025ResidualViT for Efficient Temporally Dense Video Encodingcited by 2
  10. 2025Image-to-Video Transfer Learning based on Image-Language Foundation Models: A Comprehensive Surveycited by 2
  11. 2026Frame2Freq: Spectral Adapters for Fine-Grained Video Understandingcited by 2
  12. 2023D2ST-Adapter: Disentangled-and-Deformable Spatio-Temporal Adapter for Few-shot Action Recognitioncited by 1
  13. 2024AM Flow: Adapters for Temporal Processing in Action Recognitioncited by 1
  14. 2025Reducing Annotation Burden: Exploiting Image Knowledge for Few-Shot Medical Video Object Segmentation via Spatiotemporal Consistency Relearningcited by 1
  15. 2025Advancing Video Self-Supervised Learning via Image Foundation Modelscited by 1
  16. 2026Exploring High-Order Self-Similarity for Video Understandingcited by 1
  17. 2026MamBOA: State-Space Architecture for Video Recognitioncited by 1
  18. 2024Mumpy: Multilateral Temporal-view Pyramid Transformer for Video Inpainting Detectioncited by 0
  19. 2024TDS-CLIP: Temporal Difference Side Network for Efficient VideoAction Recognitioncited by 0
  20. 2024Loose Social-Interaction Recognition in Real-world Therapy Scenarioscited by 0
  21. 2025CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasetscited by 0
  22. 2025Task-Specific Distance Correlation Matching for Few-Shot Action Recognitioncited by 0
  23. 2026EV-CLIP: Efficient Visual Prompt Adaptation for CLIP in Few-shot Action Recognition under Visual Challengescited by 0
  24. 2026PKS4:Parallel Kinematic Selective State Space Scanners for Efficient Video Understandingcited by 0
  25. 2026VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfercited by 0
  26. 2026V-LynX: Token Interface Alignment for Video+X LLMscited by 0
  27. 2026Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?cited by 0
  28. 2026TRUST: Efficient Abdominal Trauma Recognition via Image-to-Ultrasound-Video Transfer Learningcited by 0
  29. 2026Revisiting Natural Language Prompts from Large Language Models for Open-Vocabulary Video Recognitioncited by 0