mlsanityData to 7 Oct 2026

The All-Seeing Project V2: Towards General Relation Comprehension of the Open World

arXiv 2402.19474 ↗First posted 29 Feb 2024cs.CVcited by 90

Not on the map. Why?

Cited by 58 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the All-Seeing Project V2: a new model and dataset designed for understanding object relations in images. Specifically, we propose the All-Seeing Model V2 (ASMv2) that integrates the formulation of text generation, object localization, and relation comprehension into a relation conversation (ReC) task. Leveraging this unified task, our model excels not only in perceiving and recognizing all objects within the image but also in grasping the intricate relation graph between them, diminishing the relation hallucination often encountered by Multi-modal Large Language Models (MLLMs). To facilitate training and evaluation of MLLMs in relation understanding, we created the first high-quality ReC dataset ({AS-V2) which is aligned with the format of standard instruction tuning data. In addition, we design a new benchmark, termed Circular-based Relation Probing Evaluation (CRPE) for comprehensively evaluating the relation comprehension capabilities of MLLMs. Notably, our ASMv2 achieves an overall accuracy of 52.04 on this relation-aware benchmark, surpassing the 43.14 of LLaVA-1.5 by a large margin. We hope that our work can inspire more future research and contribute to the evolution towards artificial general intelligence. Our project is released at https://github.com/OpenGVLab/all-seeing.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 91 of its 110 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

91 of 110 references.

References · 91

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  13. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  15. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  16. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  17. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  18. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  19. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  20. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  21. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  22. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  23. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  24. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  25. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  26. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  27. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  28. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  29. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  30. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  31. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  32. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  33. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  34. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  35. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  36. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  37. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  38. 2017Scene Graph Generation by Iterative Message Passingcited by 462

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 32; 2025: 44; 2026: 14.

90 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  5. 2025Qwen2.5-Omni Technical Reportcited by 704
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  7. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  8. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  9. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  10. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  11. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  12. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
38 more
  1. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  2. 2024Benchmarking and Improving Detail Image Captioncited by 81
  3. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perceptioncited by 72
  4. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  5. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  6. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  7. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  8. 2024Needle In A Multimodal Haystackcited by 56
  9. 2025LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Modelscited by 53
  10. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  11. 2024The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audiocited by 48
  12. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  13. 2024ParGo: Bridging Vision-Language with Partial and Global Viewscited by 40
  14. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  15. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  16. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  17. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  18. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 34
  19. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  20. 2024Language-Image Models with 3D Understandingcited by 31
  21. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  22. 20253D Aware Region Prompted Vision Language Modelcited by 31
  23. 2024Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Modelscited by 30
  24. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  25. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  26. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  27. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 23
  28. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understandingcited by 21
  29. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 21
  30. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignmentcited by 19
  31. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  32. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  33. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  34. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  35. 2025NVIDIA Nemotron Nano V2 VLcited by 15
  36. 2024World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filteringcited by 14
  37. 2025MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal outputcited by 14
  38. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RelationVLM: Making Large Vision-Language Models Understand Visual Relations
  2. 2025ART: Adaptive Relation Tuning for Generalized Relation Prediction
  3. 2026RECAP: Relation Evidence Calibration for Detecting Spatial Relation Hallucinations in Vision-Language Models
  4. 2025Relation-R1: Progressively Cognitive Chain-of-Thought Guided Reinforcement Learning for Unified Relation Comprehension
  5. 2026RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs
  6. 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations
  7. 2026SEER: A Self-Grounded Evidence Interface for Controlled Spatial Relation Classification
  8. 2024Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation