mlsanityData to 14 Oct 2026

Joint learning of images and videos with a single Vision Transformer

arXiv 2308.10533 ↗First posted 21 Aug 2023cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In this study, we propose a method for jointly learning of images and videos using a single model. In general, images and videos are often trained by separate models. We propose in this paper a method that takes a batch of images as input to Vision Transformer IV-ViT, and also a set of video frames with temporal aggregation by late fusion. Experimental results on two image datasets and two action recognition datasets are presented.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

36 of 53 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2017The Kinetics Human Action Video Datasetcited by 1,909
  9. 2018SlowFast Networks for Video Recognitioncited by 1,867
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,143
24 more
  1. 2021Video Swin Transformercited by 1,091
  2. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  3. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  4. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 637
  5. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 387
  8. 2021Video Transformer Networkcited by 272
  9. 2018Human Action Recognition and Prediction: A Surveycited by 227
  10. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 215
  11. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  12. 2020A Comprehensive Study of Deep Video Action Recognitioncited by 88
  13. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  14. 2021Space-time Mixing Attention for Video Transformercited by 84
  15. 2020The AVA-Kinetics Localized Human Actions Video Datasetcited by 82
  16. 2019Gate-Shift Networks for Video Action Recognitioncited by 76
  17. 2022Video Transformers: A Surveycited by 63
  18. 2021Token Shift Transformer for Video Classificationcited by 61
  19. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  20. 2020Omni-sourced Webly-supervised Learning for Video Recognitioncited by 48
  21. 2021Can An Image Classifier Suffice For Action Recognition?cited by 20
  22. 2022Vision Transformer with Cross-attention by Temporal Shift for Efficient Action Recognitioncited by 5
  23. 2022Model-agnostic Multi-Domain Learning with Domain-Specific Adapters for Action Recognitioncited by 3
  24. 2019UniDual: A Unified Model for Image and Video Understandingcited by 2

Built on by

Not cited yet.