mlsanityData to 14 Oct 2026

SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs

arXiv 2603.12382 ↗First posted 12 Mar 2026cs.CV cs.AIcited by 2

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: https://risys-lab.github.io/SPARROW

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 68 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

60 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning22 Sep
  2. 2025GLUS: Global-Local Reasoning Unified into A Single Large Language Model for Video Segmentation10 Apr
  3. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2023Segment Anythingcited by 8,234
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  13. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  14. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 724
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  16. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  17. 2018GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wildcited by 475
  18. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  19. 2018LaSOT: A High-quality Benchmark for Large-scale Single Object Trackingcited by 429
  20. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  21. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  22. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  23. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 290
  24. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  25. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 228
  26. 2018Video Object Segmentation with Language Referring Expressionscited by 162
  27. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  28. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 119
  29. 2020Where Does It Exist: Spatio-Temporal Video Grounding for Multi-Form Sentencescited by 118
  30. 2018Actor and Action Video Segmentation from a Sentencecited by 106
  31. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  32. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
  33. 2023V3Det: Vast Vocabulary Visual Detection Datasetcited by 81
  34. 2019Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Videocited by 60
  35. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  36. 2024VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videoscited by 51
  37. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMscited by 47
  38. 2024Artemis: Towards Referential Understanding in Complex Videoscited by 25

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  2. 2026CheXGround: Anatomical Region Tokens for Grounded Longitudinal Chest X-ray Interpretationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PhysMLLMs: Spatial Priors for Unified Referring Segmentation and Grounded Reasoning of Images and Videos
  2. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Model
  3. 2026The Visual Bottleneck: Sparse-Frame Adaptation of MLLMs for Joint Spatial-Temporal Video Grounding
  4. 2025PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?
  5. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
  6. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning
  7. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding
  8. 2025Grounding Everything in Tokens for Multimodal Large Language Models