mlsanityData to 7 Oct 2026

Omnivore: A Single Model for Many Visual Modalities

arXiv 2201.08377 ↗First posted 20 Jan 2022cs.CV cs.AI cs.IRcited by 214

Not on the map. Why?

Abstract (arXiv)

Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our 'Omnivore' model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivore is simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. Omnivore's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.

Key facts. Cited by 214 papers in the map (data through 7 Oct 2026). Builds on 78 of its 106 references that are in the map. First posted on arXiv 20 Jan 2022.

Builds on

78 of 106 references.

References · 78

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016Layer Normalizationcited by 5,579
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
38 more
  1. 2014Going Deeper with Convolutionscited by 4,231
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  4. 2017Non-local Neural Networkscited by 2,290
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  8. 2017The Kinetics Human Action Video Datasetcited by 1,894
  9. 2018SlowFast Networks for Video Recognitioncited by 1,862
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  13. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  14. 2020Point Transformercited by 1,462
  15. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  16. 2017Random Erasing Data Augmentationcited by 1,235
  17. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  18. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  19. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  20. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  21. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  22. 2021Video Swin Transformercited by 1,090
  23. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  24. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  25. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  26. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  27. 2021Multiscale Vision Transformerscited by 896
  28. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  29. 2018Image Transformercited by 770
  30. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  31. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  32. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  33. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  34. 2016Cross-stitch Networks for Multi-task Learningcited by 466
  35. 2017Look, Listen and Learncited by 452
  36. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  37. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  38. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 43; 2023: 60; 2024: 41; 2025: 48; 2026: 22.

214 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  3. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  4. 2022Multimodal Learning with Transformers: A Surveycited by 399
  5. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  6. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  7. 2022Learning Video Representations from Large Language Modelscited by 204
  8. 2023Video Understanding with Large Language Models: A Surveycited by 203
  9. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  10. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  11. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  12. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
38 more
  1. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  2. 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 115
  3. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  4. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  5. 2022OmniMAE: Single Model Masked Pretraining on Images and Videoscited by 99
  6. 2022Towards Grand Unification of Object Trackingcited by 93
  7. 2023An Outlook into the Future of Egocentric Visioncited by 84
  8. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  9. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  10. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  11. 2024Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentationcited by 71
  12. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  13. 2024OmniSat: Self-Supervised Modality Fusion for Earth Observationcited by 63
  14. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  15. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  16. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  17. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  18. 2022Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learningcited by 49
  19. 2024Multimodal Fusion on Low-quality Data: A Comprehensive Surveycited by 48
  20. 2022Audiovisual Masked Autoencoderscited by 47
  21. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  22. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
  23. 2024Multimodal Prototyping for cancer survival predictioncited by 47
  24. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  25. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  26. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  27. 2024EgoVideo: Exploring Egocentric Foundation Model and Downstream Adaptationcited by 44
  28. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Modelscited by 42
  29. 2024GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformercited by 42
  30. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  31. 2023Dual-path Adaptation from Image to Video Transformerscited by 40
  32. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  33. 2024StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentationcited by 38
  34. 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
  35. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  36. 2023Decoupling Common and Unique Representations for Multimodal Self-supervised Learningcited by 36
  37. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  38. 2023EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understandingcited by 32

50 most cited of 214 All 214 (JSON)