mlsanityData to 7 Oct 2026

HyperSeg: Towards Universal Visual Segmentation with Large Language Model

arXiv 2411.17606 ↗First posted 26 Nov 2024cs.CVcited by 33

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

This paper aims to address universal segmentation for image and video perception with the strong reasoning ability empowered by Visual Large Language Models (VLLMs). Despite significant progress in current unified segmentation methods, limitations in adaptation to both image and video scenarios, as well as the complex reasoning segmentation, make it difficult for them to handle various challenging instructions and achieve an accurate understanding of fine-grained vision-language correlations. We propose HyperSeg, the first VLLM-based universal segmentation model for pixel-level image and video perception, encompassing generic segmentation tasks and more complex reasoning perception tasks requiring powerful reasoning abilities and world knowledge. Besides, to fully leverage the recognition capabilities of VLLMs and the fine-grained visual information, HyperSeg incorporates hybrid entity recognition and fine-grained visual perceiver modules for various segmentation tasks. Combined with the temporal adapter, HyperSeg achieves a comprehensive understanding of temporal information. Experimental results validate the effectiveness of our insights in resolving universal image and video segmentation tasks, including the more complex reasoning perception tasks. Our code is available.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 65 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

58 of 65 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos29 Sep
  2. 2024VISA: Reasoning Video Object Segmentation via Large Language Models16 Jul
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding27 Jun
  4. 2023Tracking with Human-Intent Reasoning29 Dec
  5. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  6. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2017Mask R-CNNcited by 7,259
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  5. 2016Modeling Context in Referring Expressionscited by 1,054
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2023Segment Everything Everywhere All at Oncecited by 546
  8. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  9. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  10. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  12. 2019Video Instance Segmentationcited by 391
  13. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  14. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  15. 2022OneFormer: One Transformer to Rule Universal Image Segmentationcited by 327
  16. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  17. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  18. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  19. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  20. 2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
  21. 2023A Simple Framework for Open-Vocabulary Segmentation and Detectioncited by 197
  22. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  23. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  24. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  25. 2022Open-Vocabulary Universal Image Segmentation with MaskCLIPcited by 131
  26. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  27. 2022In Defense of Online Models for Video Instance Segmentationcited by 97
  28. 2024OMG-Seg: Is One Model Good Enough For All Segmentation?cited by 73
  29. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  30. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  31. 2023PerceptionGPT: Effectively Fusing Visual Perception into LLMcited by 59
  32. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  33. 2023Spectrum-guided Multi-granularity Referring Video Object Segmentationcited by 48
  34. 2018The 2018 DAVIS Challenge on Video Object Segmentationcited by 47
  35. 2023Open-Vocabulary Semantic Segmentation with Decoupled One-Pass Networkcited by 40
  36. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  37. 2023DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Modelcited by 30
  38. 2023Universal Segmentation at Arbitrary Granularity with Language Instructioncited by 29

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 13.

33 papers.

Sort:

By area:

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  2. 2026Image Generators are Generalist Vision Learnerscited by 40
  3. 2025DisasterM3: A Remote Sensing Vision-Language Dataset for Disaster Damage Assessment and Responsecited by 35
  4. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  5. 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
  6. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoningcited by 27
  7. 2025LENS: Learning to Segment Anything with Unified Reinforced Reasoningcited by 18
  8. 2025Reinforcing Video Reasoning Segmentation to Think Before It Segmentscited by 15
  9. 2025SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Imagescited by 10
  10. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  11. 2025MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentationcited by 7
  12. 2025Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucinationcited by 7
21 more
  1. 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 6
  2. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  3. 2025Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentationcited by 5
  4. 2025LVLM_CSP: Accelerating Large Vision Language Models via Clustering, Scattering, and Pruning for Reasoning Segmentationcited by 4
  5. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  6. 2025Evolving, Not Training: Zero-Shot Reasoning Segmentation via Evolutionary Promptingcited by 4
  7. 2024Tell Me What to Track: Infusing Robust Language Guidance for Enhanced Referring Multi-Object Trackingcited by 2
  8. 2025FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentationcited by 2
  9. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Modelscited by 2
  10. 2026AgentRVOS: Reasoning over Object Tracks for Zero-Shot Referring Video Object Segmentationcited by 2
  11. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoningcited by 2
  12. 2026PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seekingcited by 2
  13. 2025LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentationcited by 1
  14. 2026X2SAM: Any Segmentation in Images and Videoscited by 1
  15. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  16. 2026FlowSeg: Dynamic Semantic Guidance for LLM-Conditioned Segmentationcited by 1
  17. 2025ComposeAnything: Composite Object Priors for Text-to-Image Generationcited by 0
  18. 2025Saliency-R1: Incentivizing Unified Saliency Reasoning Capability in MLLM with Confidence-Guided Reinforcement Learningcited by 0
  19. 2026AgentRVOS for MeViS-Text Track of 5th PVUW Challenge: 3rd Methodcited by 0
  20. 2026RCoT-Seg: Reinforced Chain-of-Thought for Video Reasoning and Segmentationcited by 0
  21. 2026VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LOGICSEG: Parsing Visual Semantics with Neural Logic Learning and Reasoning
  2. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning
  3. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
  4. 2023IFSeg: Image-free Semantic Segmentation via Vision-Language Model
  5. 2025X-SAM: From Segment Anything to Any Segmentation
  6. 2024PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation
  7. 2025ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learning
  8. 2026Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design