mlsanityData to 7 Oct 2026

The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World

arXiv 2308.01907 ↗First posted 3 Aug 2023cs.CVcited by 114

Not on the map. Why?

Cited by 61 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the All-Seeing (AS) project: a large-scale data and model for recognizing and understanding everything in the open world. Using a scalable data engine that incorporates human feedback and efficient models in the loop, we create a new dataset (AS-1B) with over 1 billion regions annotated with semantic tags, question-answering pairs, and detailed captions. It covers a wide range of 3.5 million common and rare concepts in the real world, and has 132.2 billion tokens that describe the concepts and their attributes. Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified framework for panoptic visual recognition and understanding. The model is trained with open-ended language prompts and locations, which allows it to generalize to various vision and language tasks with remarkable zero-shot performance, including region-text retrieval, region recognition, captioning, and question-answering. We hope that this project can serve as a foundation for vision-language artificial general intelligence research. Models and the dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and demo can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 87 of its 114 references that are in the map. First posted on arXiv 3 Aug 2023.

Builds on

87 of 114 references.

References · 87

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2023Segment Anythingcited by 8,158
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2017Mask R-CNNcited by 7,259
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  13. 2015VQA: Visual Question Answeringcited by 2,489
  14. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  15. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  16. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  17. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  18. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  19. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  20. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  21. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  22. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  23. 2021Grounded Language-Image Pre-trainingcited by 1,147
  24. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  25. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  26. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  27. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  28. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  29. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  30. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  31. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  32. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  33. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  34. 2018Panoptic Segmentationcited by 796
  35. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  36. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  37. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  38. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 60; 2025: 30; 2026: 7.

114 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  5. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  9. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  10. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  11. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  12. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
38 more
  1. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  2. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  3. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  4. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  5. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  6. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  7. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  8. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  9. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  10. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 85
  11. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  12. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  13. 2024Benchmarking and Improving Detail Image Captioncited by 81
  14. 2024MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizercited by 80
  15. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 79
  16. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  17. 2024RegionGPT: Towards Region Understanding Vision Language Modelcited by 71
  18. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  19. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  20. 2024Needle In A Multimodal Haystackcited by 56
  21. 2024RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesiscited by 55
  22. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  23. 2023Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Modelcited by 52
  24. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  25. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  26. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  27. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  28. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  29. 2023Myriad: Large Multimodal Model by Applying Vision Experts for Industrial Anomaly Detectioncited by 39
  30. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perceptioncited by 39
  31. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  32. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  33. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  34. 2023Tokenize Anything via Promptingcited by 31
  35. 20253D Aware Region Prompted Vision Language Modelcited by 31
  36. 2023Segment and Caption Anythingcited by 30
  37. 2024Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referringcited by 29
  38. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28

50 most cited of 114 All 114 (JSON)