mlsanityData to 14 Oct 2026

Osprey: Pixel Understanding with Visual Instruction Tuning

arXiv 2312.10032 ↗First posted 15 Dec 2023cs.CVcited by 159

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have recently achieved impressive general-purpose vision-language capabilities through visual instruction tuning. However, current MLLMs primarily focus on image-level or box-level understanding, falling short in achieving fine-grained vision-language alignment at pixel level. Besides, the lack of mask-based instruction data limits their advancements. In this paper, we propose Osprey, a mask-text instruction tuning approach, to extend MLLMs by incorporating fine-grained mask regions into language instruction, aiming at achieving pixel-wise visual understanding. To achieve this goal, we first meticulously curate a mask-based region-text dataset with 724K samples, and then design a vision-language model by injecting pixel-level representation into LLM. Specifically, Osprey adopts a convolutional CLIP backbone as the vision encoder and employs a mask-aware visual extractor to extract precise visual mask features from high resolution input. Experimental results demonstrate Osprey's superiority in various region understanding tasks, showcasing its new capability for pixel-level instruction tuning. In particular, Osprey can be integrated with Segment Anything Model (SAM) seamlessly to obtain multi-granularity semantics. The source code, dataset and demo can be found at https://github.com/CircleRadon/Osprey.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 70 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

53 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models25 Aug
  3. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2023Segment Anythingcited by 8,234
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
36 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  13. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  14. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  15. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  16. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  17. 2023Segment Anything in High Qualitycited by 376
  18. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  19. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  20. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 232
  21. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 222
  22. 2023Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIPcited by 201
  23. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 188
  24. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  25. 2023PACO: Parts and Attributes of Common Objectscited by 146
  26. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  27. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 132
  28. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  29. 2021PartImageNet: A Large, High-Quality Dataset of Partscited by 107
  30. 2022High-Quality Entity Segmentationcited by 93
  31. 2023Hierarchical Open-vocabulary Universal Image Segmentationcited by 54
  32. 2023EdgeSAM: Prompt-In-the-Loop Distillation for SAMcited by 41
  33. 2023Vocabulary-free Image Classificationcited by 35
  34. 2022Box2Mask: Box-supervised Instance Segmentation via Level-set Evolutioncited by 22
  35. 2023Label-efficient Segmentation via Affinity Propagationcited by 7
  36. 2023AIMS: All-Inclusive Multi-Level Segmentationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 51; 2025: 68; 2026: 40.

159 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  4. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  5. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 172
  6. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  7. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 152
  8. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 137
  9. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 127
  10. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 119
  11. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 101
  12. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
38 more
  1. 2024DINO-X: A Unified Vision Model for Open-World Object Detection and Understandingcited by 89
  2. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  3. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
  4. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  5. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
  6. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretrainingcited by 52
  7. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 51
  8. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
  9. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  10. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 49
  11. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 49
  12. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 46
  13. 2024Towards Semantic Equivalence of Tokenization in Multimodal LLMcited by 45
  14. 2024VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Modelscited by 43
  15. 2024Graphic Design with Large Multimodal Modelcited by 40
  16. 2024FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMscited by 38
  17. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 36
  18. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  19. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  20. 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
  21. 20253D Aware Region Prompted Vision Language Modelcited by 33
  22. 2025Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMscited by 32
  23. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  24. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 30
  25. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  26. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29
  27. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 27
  28. 2025MedSAM3: Delving into Segment Anything with Medical Conceptscited by 26
  29. 2024DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLMcited by 24
  30. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  31. 2025Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentationcited by 22
  32. 2025Multi-Layer Visual Feature Fusion in Multimodal LLMs: Methods, Analysis, and Best Practicescited by 22
  33. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  34. 2025Multimodal Referring Segmentation: A Surveycited by 22
  35. 2024The Evolution of Multimodal Model Architecturescited by 21
  36. 2024Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketchescited by 21
  37. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 21
  38. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularitycited by 21

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  2. 2023Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?
  3. 2025Visual Representation Alignment for Multimodal Large Language Models
  4. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Models
  5. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models
  6. 2024MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning
  7. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks
  8. 2026Visual Instruction Tuning Aligns Modalities through Abstraction