mlsanityData to 14 Oct 2026

MoTE: Reconciling Generalization with Specialization for Visual-Language to Video Knowledge Transfer

arXiv 2410.10589 ↗First posted 14 Oct 2024cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Transferring visual-language knowledge from large-scale foundation models for video recognition has proved to be effective. To bridge the domain gap, additional parametric modules are added to capture the temporal information. However, zero-shot generalization diminishes with the increase in the number of specialized parameters, making existing works a trade-off between zero-shot and close-set performance. In this paper, we present MoTE, a novel framework that enables generalization and specialization to be balanced in one unified model. Our approach tunes a mixture of temporal experts to learn multiple task views with various degrees of data fitting. To maximally preserve the knowledge of each expert, we propose Weight Merging Regularization, which regularizes the merging process of experts in weight space. Additionally with temporal feature modulation to regularize the contribution of temporal feature during test. We achieve a sound balance between zero-shot and close-set video recognition tasks and obtain state-of-the-art or competitive results on various datasets, including Kinetics-400 & 600, UCF, and HMDB. Code is available at urlhttps://github.com/ZMHH-H/MoTE.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 54 of its 59 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

54 of 59 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  6. 2017The Kinetics Human Action Video Datasetcited by 1,909
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2021ViViT: A Video Vision Transformercited by 1,620
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
38 more
  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  2. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  4. 2021Multiscale Vision Transformerscited by 897
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  6. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  7. 2022Language-driven Semantic Segmentationcited by 717
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2021Editing Factual Knowledge in Language Modelscited by 553
  10. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  11. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  12. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  13. 2018A Short Note about Kinetics-600cited by 333
  14. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  15. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  16. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  17. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
  18. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 203
  19. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  20. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 179
  21. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 148
  22. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  23. 2023CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matchingcited by 133
  24. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 113
  25. 2023Vita-CLIP: Video and text adaptive CLIP via Multimodal Promptingcited by 91
  26. 2017Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapescited by 84
  27. 2021Elaborative Rehearsal for Zero-shot Action Recognitioncited by 77
  28. 2020Why Do Better Loss Functions Lead to Less Transferable Features?cited by 73
  29. 2022Revisiting Classifier: Transferring Vision-Language Models for Video Recognitioncited by 71
  30. 2023Open-VCLIP: Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimizationcited by 55
  31. 2022Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Modelscited by 49
  32. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 46
  33. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  34. 2024FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognitioncited by 36
  35. 2023Towards Calibrated Robust Fine-Tuning of Vision-Language Modelscited by 35
  36. 2023Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learningcited by 32
  37. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Modelscited by 31
  38. 2022No Reason for No Supervision: Improved Generalization in Supervised Modelscited by 23

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 3.

5 papers.

  1. 2025CleanPose: Category-Level Object Pose Estimation via Causal Learning and Knowledge Distillationcited by 10
  2. 2024Efficient Transfer Learning for Video-language Foundation Modelscited by 4
  3. 2026MemPose: Category-level Object Pose Estimation with Memorycited by 1
  4. 2026VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfercited by 0
  5. 2026TACO: Towards Task-Consistent Open-Vocabulary Adaptation in Video Recognitioncited by 0