mlsanityData to 7 Oct 2026

From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs

arXiv 2603.17228 ↗First posted 18 Mar 2026cs.CV cs.AI cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood. We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM. We further conduct an intervention based attention knockout analysis to test whether cross-token attention progressively refines visual representations, and an evaluation of bidirectional attention among image tokens on spatial consistency. Our analysis reveals that the adapter introduces a segmentation representation drop-off, but LLM layers progressively recover through attention-mediated refinement, where correctly classified tokens steer misclassified neighbors toward the correct label. At early image token positions, this recovery is bounded by causal attention, which bidirectional attention among image tokens alleviates. These findings provide a mechanistic account of how MLLMs process visual information for segmentation, informing the design of future segmentation-capable models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 46 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

41 of 46 references.

References · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  7. 2017Mask R-CNNcited by 7,259
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
29 more
  1. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  5. 2025SAM 3: Segment Anything with Conceptscited by 835
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  11. 2023Segment Everything Everywhere All at Oncecited by 546
  12. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  13. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  14. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  15. 2016Segmentation from Natural Language Expressionscited by 257
  16. 2023PixelLM: Pixel Reasoning with Large Multimodal Modelcited by 220
  17. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  18. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  19. 2022Linearly Mapping from Image to Text Spacecited by 139
  20. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  21. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  22. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  23. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  24. 2024GROUNDHOG: Grounding Large Language Models to Holistic Segmentationcited by 70
  25. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  26. 2024Dense Connector for MLLMscited by 48
  27. 2023A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineeringcited by 45
  28. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 33
  29. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Recovery to Drop-off: How Action Post-training Reduces a VLM's Late-Layer Depth Decodabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking MLLM Itself as a Segmenter with a Single Segmentation Token
  2. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMs
  3. 2026Segmentation before Answering: Pixel Grounding for MLLM Visual Reasoning
  4. 2025Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference
  5. 2025HERO: Rethinking Visual Token Early Dropping in High-Resolution Large Vision-Language Models
  6. 2026Large Vision-Language Models Get Lost in Attention
  7. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs
  8. 2026Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStack