mlsanityData to 7 Oct 2026

One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

arXiv 2409.19603 ↗First posted 29 Sep 2024cs.CV cs.AIcited by 99

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of large language models, and augmented by the Segment Anything Model, VideoLISA generates temporally consistent segmentation masks in videos based on language instructions. Existing image-based methods, such as LISA, struggle with video tasks due to the additional temporal dimension, which requires temporal dynamic understanding and consistent segmentation across frames. VideoLISA addresses these challenges by integrating a Sparse Dense Sampling strategy into the video-LLM, which balances temporal context and spatial detail within computational constraints. Additionally, we propose a One-Token-Seg-All approach using a specially designed <TRK> token, enabling the model to segment and track objects across multiple frames. Extensive evaluations on diverse benchmarks, including our newly introduced ReasonVOS benchmark, demonstrate VideoLISA's superior performance in video object segmentation tasks involving complex reasoning, temporal understanding, and object tracking. While optimized for videos, VideoLISA also shows promising generalization to image segmentation, revealing its potential as a unified foundation model for language-instructed object segmentation. Code and model will be available at: https://github.com/showlab/VideoLISA.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 63 of its 74 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

63 of 74 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Tracking with Human-Intent Reasoning29 Dec
  2. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  4. 2023DetGPT: Detect What You Need via Reasoning23 May

Other references · 59

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Segment Anythingcited by 8,158
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2021Video Swin Transformercited by 1,090
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
38 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  2. 2023A Survey on Multimodal Large Language Modelscited by 898
  3. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  4. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  5. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  6. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  7. 2016Fully-Convolutional Siamese Networks for Object Trackingcited by 566
  8. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  9. 2023Segment Everything Everywhere All at Oncecited by 546
  10. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  11. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  12. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  13. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  14. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  15. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  16. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  17. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  18. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  19. 2019Cross-Modal Self-Attention Network for Referring Image Segmentationcited by 263
  20. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  21. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  22. 2023MOSE: A New Dataset for Video Object Segmentation in Complex Scenescited by 232
  23. 2014Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Partscited by 230
  24. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  25. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  26. 2022Language as Queries for Referring Video Object Segmentationcited by 180
  27. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  28. 2023PACO: Parts and Attributes of Common Objectscited by 146
  29. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  30. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  31. 2018Actor and Action Video Segmentation from a Sentencecited by 105
  32. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  33. 2022BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Videocited by 66
  34. 2023OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentationcited by 62
  35. 2023SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentationcited by 57
  36. 2024Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentationcited by 55
  37. 2023XMem++: Production-level Video Segmentation From Few Annotated Framescited by 53
  38. 2022Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentationcited by 50

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 44; 2026: 47.

99 papers.

Sort:

By area:

  1. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 234
  2. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  3. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  4. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 83
  5. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  6. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  7. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  8. 2024SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentationcited by 43
  9. 2025VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generationcited by 39
  10. 2024HyperSeg: Towards Universal Visual Segmentation with Large Language Modelcited by 33
  11. 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learningcited by 31
  12. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
38 more
  1. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  2. 2025PixelThink: Towards Efficient Chain-of-Pixel Reasoningcited by 28
  3. 2025The Devil is in Temporal Token: High Quality Video Reasoning Segmentationcited by 26
  4. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 24
  5. 2025Multimodal Referring Segmentation: A Surveycited by 22
  6. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  7. 2024UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editingcited by 18
  8. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  9. 2025Impossible Videoscited by 16
  10. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentationcited by 16
  11. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  12. 2025VideoMolmo: Spatio-Temporal Grounding Meets Pointingcited by 14
  13. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  14. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  15. 2025Object-centric Video Question Answering with Visual Grounding and Referringcited by 12
  16. 2026WildDet3D: Scaling Promptable 3D Detection in the Wildcited by 12
  17. 2025Fine-grained Spatiotemporal Grounding on Egocentric Videoscited by 11
  18. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 11
  19. 2025Towards Omnimodal Expressions and Reasoning in Referring Audio-Visual Segmentationcited by 10
  20. 2025Deforming Videos to Masks: Flow Matching for Referring Video Segmentationcited by 10
  21. 2025RVTBench: A Benchmark for Visual Reasoning Taskscited by 9
  22. 2025CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videoscited by 8
  23. 2025SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Modelscited by 8
  24. 2026Refer-Agent: A Collaborative Multi-Agent System with Reasoning and Reflection for Referring Video Object Segmentationcited by 7
  25. 2025LangGas: Introducing Language in Selective Zero-Shot Background Subtraction for Semi-Transparent Gas Leak Detection with a New Datasetcited by 6
  26. 2025InterRVOS: Interaction-aware Referring Video Object Segmentationcited by 6
  27. 2026LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generationcited by 6
  28. 2026InstructSAM: Segment Any Instance with Any Instructionscited by 6
  29. 2026VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understandingcited by 5
  30. 2025BrokenVideos: A Benchmark Dataset for Fine-Grained Artifact Localization in AI-Generated Videoscited by 4
  31. 2025MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understandingcited by 4
  32. 2026Instance-Aligned Captions for Explainable Video Anomaly Detectioncited by 4
  33. 2024Referring Video Object Segmentation via Language-aligned Track Selectioncited by 3
  34. 2025SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Groundingcited by 3
  35. 20263AM: 3egment Anything with Geometric Consistency in Videoscited by 3
  36. 2025A Comprehensive Survey on Video Scene Parsing:Advances, Challenges, and Prospectscited by 2
  37. 2025HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentationcited by 2
  38. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024VISA: Reasoning Video Object Segmentation via Large Language Models
  2. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models
  3. 2026MoVISA: Multi-Token Reasoning for Video Object Segmentation
  4. 2024ViLLa: Video Reasoning Segmentation with Large Language Model
  5. 2025EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Models
  6. 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search
  7. 2024LLM-Seg: Bridging Image Segmentation and Large Language Model Reasoning
  8. 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation