mlsanityData to 7 Oct 2026

ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

arXiv 2305.11172 ↗First posted 18 May 2023cs.CV cs.CL cs.SDcited by 117

Not on the map. Why?

Cited by 28 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we explore a scalable way for building a general representation model toward unlimited modalities. We release ONE-PEACE, a highly extensible model with 4B parameters that can seamlessly align and integrate representations across vision, audio, and language modalities. The architecture of ONE-PEACE comprises modality adapters, shared self-attention layers, and modality FFNs. This design allows for the easy extension of new modalities by adding adapters and FFNs, while also enabling multi-modal fusion through self-attention layers. To pretrain ONE-PEACE, we develop two modality-agnostic pretraining tasks, cross-modal aligning contrast and intra-modal denoising contrast, which align the semantic space of different modalities and capture fine-grained details within modalities concurrently. With the scaling-friendly architecture and pretraining tasks, ONE-PEACE has the potential to expand to unlimited modalities. Without using any vision or language pretrained model for initialization, ONE-PEACE achieves leading results on a wide range of uni-modal and multi-modal tasks, including image classification (ImageNet), semantic segmentation (ADE20K), audio-text retrieval (AudioCaps, Clotho), audio classification (ESC-50, FSD50K, VGGSound), audio question answering (AVQA), image-text retrieval (MSCOCO, Flickr30K), and visual grounding (RefCOCO/+/g). Code is available at https://github.com/OFA-Sys/ONE-PEACE.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 149 of its 172 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

149 of 172 references.

References · 149

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  13. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  14. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  15. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  16. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  17. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  18. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  19. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  20. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  21. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  22. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  23. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  24. 2015VQA: Visual Question Answeringcited by 2,489
  25. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  26. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  27. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  28. 2018Deep contextualized word representationscited by 2,285
  29. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  30. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  31. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  32. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  33. 2020GLU Variants Improve Transformercited by 1,724
  34. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  35. 2021ViViT: A Video Vision Transformercited by 1,610
  36. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  37. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  38. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305

50 most cited of 149 All 149 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 50; 2025: 33; 2026: 9.

117 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  4. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  5. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  12. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
38 more
  1. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  4. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  5. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  6. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 63
  7. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  8. 2024Toward Visual Grounding: A Surveycited by 52
  9. 2023Text-image Alignment for Diffusion-based Perceptioncited by 42
  10. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  11. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  12. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  13. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35
  14. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  15. 2024TextHawk2: A Large Vision-Language Model Excels in Bilingual OCR and Grounding with 16x Fewer Tokenscited by 32
  16. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  17. 2023ViT-Lens: Towards Omni-modal Representationscited by 29
  18. 2024OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modelingcited by 28
  19. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  20. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  21. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  22. 2024FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusioncited by 19
  23. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 19
  24. 2023Audio Generation with Multiple Conditional Diffusion Modelcited by 18
  25. 2025A Survey on Remote Sensing Foundation Models: From Vision to Multimodalitycited by 18
  26. 2025RefDrone: A Challenging Benchmark for Referring Expression Comprehension in Drone Scenescited by 17
  27. 2023Rejuvenating image-GPT as Strong Visual Representation Learnerscited by 16
  28. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  29. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  30. 2024Multi-Agent VQA: Exploring Multi-Agent Foundation Models in Zero-Shot Visual Question Answeringcited by 15
  31. 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Datacited by 15
  32. 2023Low-Resolution Self-Attention for Semantic Segmentationcited by 12
  33. 2024Unified Static and Dynamic Network: Efficient Temporal Filtering for Video Groundingcited by 12
  34. 2024Siamese Vision Transformers are Scalable Audio-visual Learnerscited by 12
  35. 2024M3-JEPA: Multimodal Alignment via Multi-gate MoE based on the Joint-Embedding Predictive Architecturecited by 12
  36. 2024Dense Audio-Visual Event Localization under Cross-Modal Consistency and Multi-Temporal Granularity Collaborationcited by 12
  37. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  38. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Modelscited by 11

50 most cited of 117 All 117 (JSON)