mlsanityData to 13 Oct 2026

OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

arXiv 2107.00249 ↗First posted 1 Jul 2021cs.CVcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio multi-modal representations and achieve promising results on a variety of cross-modal understanding and generation tasks.

Key facts. Cited by 20 papers in the map (data through 13 Oct 2026). Builds on 44 of its 50 references that are in the map. First posted on arXiv 1 Jul 2021.

Builds on

44 of 50 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,712
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  8. 2016Densely Connected Convolutional Networkscited by 6,707
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  10. 2016Layer Normalizationcited by 5,624
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,284
32 more
  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  4. 2015VQA: Visual Question Answeringcited by 2,495
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,551
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  9. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  10. 2016Generative Adversarial Text to Image Synthesiscited by 735
  11. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  12. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  13. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 555
  14. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  15. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  16. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  17. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  18. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 346
  19. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  20. 2016Image Captioning with Semantic Attentioncited by 260
  21. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 227
  22. 2019Connecting Vision and Language with Localized Narrativescited by 225
  23. 2016End-to-end Learning of Deep Visual Representations for Image Retrievalcited by 122
  24. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  25. 2021Multi-Format Contrastive Learning of Audio Representationscited by 39
  26. 2018Learning Speaker Representations with Mutual Informationcited by 26
  27. 2017Transfer Learning for Speech Recognition on a Budgetcited by 25
  28. 2020Direct Speech-to-image Translationcited by 12
  29. 2020Listen, Look and Deliberate: Visual context-aware speech recognition using pre-trained text-video representationscited by 10
  30. 2018End-to-End Multimodal Speech Recognitioncited by 9
  31. 2020S2IGAN: Speech-to-Image Generation via Adversarial Learningcited by 8
  32. 2018Improving speech emotion recognition via Transformer-based Predictive Coding through transfer learningcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 10; 2024: 3; 2025: 1.

20 papers.

Sort:

By area:

  1. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  2. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  5. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 75
  6. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  7. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  8. 2023LLMDet: A Third Party Large Language Models Generated Text Detection Toolcited by 44
  9. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  10. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 42
  11. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
  12. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 25
8 more
  1. 2023Accommodating Audio Modality in CLIP for Multimodal Processingcited by 18
  2. 2025OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learningcited by 13
  3. 2023LANISTR: Multimodal Learning from Structured and Unstructured Datacited by 11
  4. 2024Pedestrian Attribute Recognition: A New Benchmark Dataset and A Large Language Model Augmented Frameworkcited by 10
  5. 2022Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASRcited by 7
  6. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategiescited by 5
  7. 2022HiVLP: Hierarchical Vision-Language Pre-Training for Fast Image-Text Retrievalcited by 4
  8. 2023Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representationcited by 4