mlsanityData to 7 Oct 2026

OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding

arXiv 2406.19389 ↗First posted 27 Jun 2024cs.CVcited by 126

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Current universal segmentation methods demonstrate strong capabilities in pixel-level image and video understanding. However, they lack reasoning abilities and cannot be controlled via text instructions. In contrast, large vision-language multimodal models exhibit powerful vision-based conversation and reasoning capabilities but lack pixel-level understanding and have difficulty accepting visual prompts for flexible user interaction. This paper proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level vision understanding with reasoning abilities. It can accept various visual and text prompts for flexible user interaction. Specifically, we use a universal segmentation method as the visual encoder, integrating image information, perception priors, and visual prompts into visual tokens provided to the LLM. The LLM is responsible for understanding the user's text instructions and providing text responses and pixel-level segmentation results based on the visual information. We propose perception prior embedding to better integrate perception priors with image features. OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model, matching or surpassing the performance of specialized methods on multiple benchmarks. Rather than using LLM to connect each specialist, our work aims at end-to-end training on one encoder, one decoder, and one LLM. The code and model have been released for further research.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 123 of its 139 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

123 of 139 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition7 May
  2. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning12 Apr
  3. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model28 Dec
  4. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  5. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep

Other references · 116

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Segment Anythingcited by 8,158
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2020End-to-End Object Detection with Transformerscited by 7,384
  11. 2017Mask R-CNNcited by 7,259
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022A ConvNet for the 2020scited by 3,545
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  7. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  8. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  15. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  16. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  17. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  18. 2016Modeling Context in Referring Expressionscited by 1,054
  19. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  20. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  21. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  22. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  23. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  24. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  25. 2018Panoptic Segmentationcited by 796
  26. 2016A Diagram Is Worth A Dozen Imagescited by 793
  27. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  28. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  29. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  30. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  31. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  32. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  33. 2019Hybrid Task Cascade for Instance Segmentationcited by 479
  34. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  35. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  36. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  37. 2024InternLM2 Technical Reportcited by 391
  38. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 70; 2026: 36.

126 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  3. 2024SIDA: Social Media Image Deepfake Detection, Localization and Explanation with Large Multimodal Modelcited by 107
  4. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  5. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  6. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  7. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  8. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  9. 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learningcited by 43
  10. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  11. 2025POSTA: A Go-to Framework for Customized Artistic Poster Generationcited by 36
  12. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
38 more
  1. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  2. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  3. 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
  4. 2024RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Expertscited by 29
  5. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  6. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  7. 2024EventGPT: Event Stream Understanding with Multimodal Large Language Modelscited by 27
  8. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  9. 2024FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularitycited by 26
  10. 2024ViLLa: Video Reasoning Segmentation with Large Language Modelcited by 23
  11. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understandingcited by 22
  12. 2025Multimodal Referring Segmentation: A Surveycited by 22
  13. 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
  14. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  15. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  16. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  17. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  18. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  19. 2025PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Modelcited by 16
  20. 2025Slot Attention with Re-Initialization and Self-Distillationcited by 14
  21. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  22. 2024Pseudo-RIS: Distinctive Pseudo-supervision Generation for Referring Image Segmentationcited by 13
  23. 2025Region-based Cluster Discrimination for Visual Representation Learningcited by 13
  24. 2025Vector-Quantized Vision Foundation Models for Object-Centric Learningcited by 11
  25. 2025UniBiomed: A Universal Foundation Model for Grounded Biomedical Image Interpretationcited by 10
  26. 2025Describe Anything in Medical Imagescited by 10
  27. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  28. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  29. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  30. 2024Multi-modal Generative AI: Multi-modal LLMs, Diffusions, and the Unificationcited by 8
  31. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  32. 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 8
  33. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  34. 2025Text4Seg++: Advancing Image Segmentation via Generative Language Modelingcited by 8
  35. 2025Pixel-Level Reasoning Segmentation via Multi-turn Conversationscited by 7
  36. 2025Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understandingcited by 7
  37. 2025The 1st Solution for 7th LSVOS RVOS Track: SaSaSa2VAcited by 7
  38. 2025SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenescited by 7

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
  2. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation
  3. 2026Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
  4. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  5. 2026Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models
  6. 2025VGR: Visual Grounded Reasoning
  7. 2024TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings
  8. 2023VCoder: Versatile Vision Encoders for Multimodal Large Language Models