mlsanityData to 7 Oct 2026

PolyViT: Co-training Vision Transformers on Images, Videos and Audio

arXiv 2111.12993 ↗First posted 25 Nov 2021cs.CV cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters? We present PolyViT, a model trained on image, audio and video which answers this question. By co-training different tasks on a single modality, we are able to improve the accuracy of each individual task and achieve state-of-the-art results on 5 standard video- and audio-classification datasets. Co-training PolyViT on multiple modalities and tasks leads to a model that is even more parameter-efficient, and learns representations that generalize across multiple domains. Moreover, we show that co-training is simple and practical to implement, as we do not need to tune hyperparameters for each combination of datasets, but can simply adapt those from standard, single-task training.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 73 of its 87 references that are in the map. First posted on arXiv 25 Nov 2021.

Builds on

73 of 87 references.

References · 73

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2017Mask R-CNNcited by 7,259
  9. 2016Layer Normalizationcited by 5,579
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2017Non-local Neural Networkscited by 2,290
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021ViViT: A Video Vision Transformercited by 1,610
  9. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  10. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,345
  11. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  14. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  15. 2016Deep Networks with Stochastic Depthcited by 973
  16. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  17. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  18. 2021Multiscale Vision Transformerscited by 896
  19. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  20. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  21. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  22. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  23. 2020X3D: Expanding Architectures for Efficient Video Recognitioncited by 635
  24. 2021AST: Audio Spectrogram Transformercited by 634
  25. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  26. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  27. 2017Self-Normalizing Neural Networkscited by 591
  28. 2017Learning multiple visual domains with residual adapterscited by 553
  29. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  30. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  31. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  32. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  33. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  34. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  35. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 419
  36. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  37. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  38. 2017Temporal Relational Reasoning in Videoscited by 324

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 18; 2024: 11; 2025: 8; 2026: 1.

58 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  4. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  5. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  6. 2023Audio-Visual LLM for Video Understandingcited by 76
  7. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  8. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  9. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  10. 2022Audiovisual Masked Autoencoderscited by 47
  11. 2022High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learningcited by 31
  12. 2022Effective Adaptation in Multi-Task Co-Training for Unified Autonomous Drivingcited by 30
38 more
  1. 2023DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Modelcited by 30
  2. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  3. 2023Towards Universal Image Embeddings: A Large-Scale Dataset and Challenge for Generic Image Representationscited by 28
  4. 2022Learning Model Predictive Controllers with Real-Time Attention for Real-World Navigationcited by 27
  5. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  6. 2022UAVM: Towards Unifying Audio and Visual Modelscited by 20
  7. 2024XTrack: Multimodal Training Boosts RGB-X Video Object Trackerscited by 20
  8. 2025RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learningcited by 20
  9. 2022TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalitiescited by 19
  10. 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perceptioncited by 19
  11. 2022HiP: Hierarchical Perceivercited by 15
  12. 2023ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoderscited by 14
  13. 2024A Versatile Framework for Multi-scene Person Re-identificationcited by 14
  14. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  15. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 12
  16. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  17. 2022Multi-dataset Training of Transformers for Robust Action Recognitioncited by 10
  18. 2022Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonizationcited by 9
  19. 2023CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasetscited by 8
  20. 2023Visual Exemplar Driven Task-Prompting for Unified Perception in Autonomous Drivingcited by 8
  21. 2023Video Task Decathlon: Unifying Image and Video Tasks in Autonomous Drivingcited by 8
  22. 2024Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learningcited by 8
  23. 2024UDON: Universal Dynamic Online distillatioN for generic image representationscited by 8
  24. 2022X-Learner: Learning Cross Sources and Tasks for Universal Visual Representationcited by 6
  25. 2023Modality-invariant Visual Odometry for Embodied Visioncited by 6
  26. 2022Play It Back: Iterative Attention for Audio Recognitioncited by 3
  27. 2024Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Losscited by 3
  28. 2025Exploring Audio Cues for Enhanced Test-Time Video Model Adaptationcited by 3
  29. 2022Deep Architectures for Content Moderation and Movie Content Ratingcited by 2
  30. 2023BenchMD: A Benchmark for Unified Learning on Medical Images and Sensorscited by 2
  31. 2023TMT-VIS: Taxonomy-aware Multi-dataset Joint Training for Video Instance Segmentationcited by 2
  32. 2025textttAVROBUSTBENCH: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Timecited by 2
  33. 2023Does Visual Pretraining Help End-to-End Reasoning?cited by 1
  34. 2023ViT-Lens: Initiating Omni-Modal Exploration through 3D Insightscited by 1
  35. 2024Computer vision tasks for intelligent aerospace missions: An overviewcited by 1
  36. 2025Lightweight Multimodal Artificial Intelligence Framework for Maritime Multi-Scene Recognitioncited by 1
  37. 2025AMA-SAM: Adversarial Multi-Domain Alignment of Segment Anything Model for High-Fidelity Histology Nuclei Segmentationcited by 1
  38. 2025Separating Shared and Domain-Specific LoRAs for Multi-Domain Learningcited by 1

50 most cited of 58 All 58 (JSON)