mlsanityData to 7 Oct 2026

AIM: Adapting Image Models for Efficient Video Action Recognition

arXiv 2302.03024 ↗First posted 6 Feb 2023cs.CVcited by 145

Not on the map. Why?

Abstract (arXiv)

Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, full finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation to gradually equip an image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters on four video action recognition benchmarks. Thanks to its simplicity, our method is also generally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage is urlhttps://adapt-image-models.github.io/.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 71 of its 79 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

71 of 79 references.

References · 71

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  5. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2021ViViT: A Video Vision Transformercited by 1,610
  10. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  13. 2017Random Erasing Data Augmentationcited by 1,235
  14. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  15. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  16. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  17. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  18. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  19. 2021Video Swin Transformercited by 1,090
  20. 2021Scaling Vision Transformerscited by 993
  21. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  22. 2021Multiscale Vision Transformerscited by 896
  23. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  24. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  25. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  26. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  27. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  28. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  29. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  30. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  31. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  32. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  33. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  34. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  35. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  36. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  37. 2019A Short Note on the Kinetics-700 Human Action Datasetcited by 308
  38. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 48; 2024: 45; 2025: 38; 2026: 14.

145 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023Segment Anything in High Qualitycited by 373
  3. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  4. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  5. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  6. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  7. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  8. 2023SAM-Med3D: Towards General-purpose Segmentation Models for Volumetric Medical Imagescited by 101
  9. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 85
  10. 20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
  11. 2024CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognitioncited by 61
  12. 2024Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognitioncited by 49
38 more
  1. 2023End-to-End Temporal Action Detection with 1B Parameters Across 1000 Framescited by 47
  2. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  3. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 44
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  5. 2023From Static to Dynamic: Adapting Landmark-Aware Image Models for Facial Expression Recognition in Videoscited by 39
  6. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 35
  7. 2023Visual Tuningcited by 34
  8. 2024FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERscited by 30
  9. 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 27
  10. 2025Video-STAR: Reinforcing Open-Vocabulary Action Recognition with Toolscited by 23
  11. 2024Mitigating the Human-Robot Domain Discrepancy in Visual Pre-training for Robotic Manipulationcited by 22
  12. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  13. 2024Distilling Vision-Language Models on Millions of Videoscited by 20
  14. 2023Diversifying Spatial-Temporal Perception for Video Domain Generalizationcited by 19
  15. 2023Promise:Prompt-driven 3D Medical Image Segmentation Using Pretrained Image Foundation Modelscited by 18
  16. 2024Region-Based Representations Revisitedcited by 18
  17. 2023STA-VPT: SpatioTemporally Aligned Visual Prompt Tuningcited by 17
  18. 2024Less Could Be Better: Parameter-efficient Fine-tuning Advances Medical Vision Foundation Modelscited by 17
  19. 2024PA-SAM: Prompt Adapter SAM for High-Quality Image Segmentationcited by 17
  20. 2026Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perceptioncited by 17
  21. 2023OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognitioncited by 16
  22. 2024XOV-Action: Towards Generalizable Open-Vocabulary Action Recognitioncited by 15
  23. 2023A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognitioncited by 14
  24. 2023ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adaptercited by 14
  25. 2023TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scalecited by 14
  26. 2023VideoGLUE: Video General Understanding Evaluation of Foundation Modelscited by 14
  27. 2023ZeroI2V: Zero-Cost Adaptation of Pre-trained Transformers from Image to Videocited by 14
  28. 2023MACP: Efficient Model Adaptation for Cooperative Perceptioncited by 14
  29. 2023EZ-CLIP: Efficient Zeroshot Video Action Recognitioncited by 14
  30. 2024OmniCLIP: Adapting CLIP for Video Recognition with Spatial-Temporal Omni-Scale Feature Learningcited by 14
  31. 2025SelaVPR++: Towards Seamless Adaptation of Foundation Models for Efficient Place Recognitioncited by 14
  32. 2023Align before Adapt: Leveraging Entity-to-Region Alignments for Generalizable Video Action Recognitioncited by 12
  33. 2024VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Modelcited by 12
  34. 2025Self-Prompt SAM: Medical Image Segmentation via Automatic Prompt SAM Adaptationcited by 12
  35. 2023A Survey of Deep Learning in Sports Applications: Perception, Comprehension, and Decisioncited by 11
  36. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  37. 2023Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learningcited by 11
  38. 2023MinD-3D: Reconstruct High-quality 3D objects in Human Braincited by 11

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning
  2. 2026Towards Data-Efficient Video Pre-training with Frozen Image Foundation Models
  3. 2023Probabilistic Adaptation of Text-to-Video Models
  4. 2024Enhancing Video Transformers for Action Understanding with VLM-aided Training
  5. 2026Efficient Spatial-Temporal Focal Adapter with SSM for Temporal Action Detection
  6. 2026Edit2Interp: Adapting Image Foundation Models from Spatial Editing to Video Frame Interpolation with Few-Shot Learning
  7. 2024M2-CLIP: A Multimodal, Multi-task Adapting Framework for Video Action Recognition
  8. 2024VideoMamba: Spatio-Temporal Selective State Space Model