LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Large Multimodal Models (LMMs) have achieved remarkable progress in general-purpose vision--language understanding, yet they remain limited in tasks requiring precise object-level grounding, fine-grained spatial reasoning, and controllable visual manipulation. In particular, existing systems often struggle to identify the correct instance, preserve object identity across interactions, and localize or modify designated regions with high precision. Object-centric vision provides a principled framework for addressing these challenges by promoting explicit representations and operations over visual entities, thereby extending multimodal systems from global scene understanding to object-level understanding, segmentation, editing, and generation. This paper presents a comprehensive review of recent advances at the convergence of LMMs and object-centric vision. We organize the literature into four major themes: object-centric visual understanding, object-centric referring segmentation, object-centric visual editing, and object-centric visual generation. We further summarize the key modeling paradigms, learning strategies, and evaluation protocols that support these capabilities. Finally, we discuss open challenges and future directions, including robust instance permanence, fine-grained spatial control, consistent multi-step interaction, unified cross-task modeling, and reliable benchmarking under distribution shift. We hope this paper provides a structured perspective on the development of scalable, precise, and trustworthy object-centric multimodal systems.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 234 of its 246 references that are in the map. First posted on arXiv 13 Apr 2026.
Builds on
234 of 246 references.
Closest ancestors · 24
Same direction, newest first
- 2026Scientific Graphics Program Synthesis via Dual Self-Consistency Reinforcement Learning7 Apr
- 2026MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods29 Jan
- 2025Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentation22 Oct
- 2025CoPRS: Learning Positional Prior from Chain-of-Thought for Reasoning Segmentation13 Oct
- 2025Video models are zero-shot learners and reasoners24 Sep
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug
- 2025Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder28 Jun
- 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning27 Jun
- 2025RSVP: Reasoning Segmentation via Visual Prompting and Multi-modal Chain-of-Thought4 Jun
- 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoning29 May
- 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning17 May
- 2025GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation10 Apr
12 more
- 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentation1 Apr
- 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement9 Mar
- 2025Pixel-Level Reasoning Segmentation via Multi-turn Conversations13 Feb
- 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentation15 Jan
- 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation19 Dec
- 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Model26 Nov
- 2024SegLLM: Multi-round Reasoning Segmentation24 Oct
- 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos29 Sep
- 2024VISA: Reasoning Video Object Segmentation via Large Language Models16 Jul
- 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding27 Jun
- 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
- 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
Other references · 210
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2017Neural Discrete Representation Learningcited by 4,712
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
38 more
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2023Qwen Technical Reportcited by 3,344
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
- 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,290
- 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,219
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
- 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
- 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
- 2022MaskGIT: Masked Generative Image Transformercited by 1,099
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
- 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
- 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
- 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 843
50 most cited of 210 All 210 (JSON)
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
- 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policiescited by 4
- 2026CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmarkcited by 3
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysiscited by 2
- 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoningcited by 1
- 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMscited by 0
- 2026MMVistaReason: Toward Open-Data and Post-Training Recipes for Multimodal Reasoning1 Oct 2026
- 2026VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies28 May 2026
- 2026InstructSAM: Segment Any Instance with Any Instructions25 May 2026
- 2026Regulating Anatomy-Aware Rewards via Trajectory-Integral Feedback for Volumetric Computed Tomography Analysis19 May 2026
- 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs19 May 2026
- 2026CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark18 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Object-Uni: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation
- 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
- 2025PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding
- 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models
- 2024Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models
- 2026MultiModal Code-Switching: Interleaving Visual Objects into Language for Explicit Object-Level Alignment
- 2026Perception-Aware Multimodal Spatial Reasoning from Monocular Images
- 2024Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models