mlsanityData to 7 Oct 2026

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

arXiv 2202.03052 ↗First posted 7 Feb 2022cs.CV cs.CLcited by 742

Not on the map. Why?

Cited by 223 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only 20M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.

Key facts. Cited by 742 papers in the map (data through 7 Oct 2026). Builds on 101 of its 123 references that are in the map. First posted on arXiv 7 Feb 2022.

Builds on

101 of 123 references.

References · 101

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2016Layer Normalizationcited by 5,579
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  13. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  14. 2016Improved Techniques for Training GANscited by 2,938
  15. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  16. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  17. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  18. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  19. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  20. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  21. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  22. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  23. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  24. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  25. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  26. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  27. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  28. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  29. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  30. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  31. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  32. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  33. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  34. 2017Random Erasing Data Augmentationcited by 1,235
  35. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  36. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  37. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  38. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 79; 2023: 303; 2024: 221; 2025: 105; 2026: 34.

742 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2022Planning-oriented Autonomous Drivingcited by 1,068
  8. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
38 more
  1. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  4. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  5. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  8. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  9. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  10. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  13. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  14. 2022Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)cited by 396
  15. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  16. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  17. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  18. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  19. 2023WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentationcited by 311
  20. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  21. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  22. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  23. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  24. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  25. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  26. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  27. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  28. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  29. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  30. 2023Text2Tex: Text-driven Texture Synthesis via Diffusion Modelscited by 234
  31. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  32. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  33. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  34. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  35. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  36. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  37. 2023SegGPT: Segmenting Everything In Contextcited by 193
  38. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190

50 most cited of 742 All 742 (JSON)