mlsanityData to 14 Oct 2026

Do Visual Grounding Decoders Need Feed-Forward Networks? A Controlled Study over Frozen Vision-Language Features

arXiv 2608.15061 ↗First posted 15 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Do feed-forward networks (FFNs) in visual grounding decoders add essential computation once a pretrained vision-language model has already encoded image and language context? We compare a four-block attention-only decoder (A4), a matched four-block attention-plus-FFN decoder (S4), and an eight-block attention-only parameter control (A8) over frozen VLM features. A4 matches or slightly exceeds S4 on RefCOCOg and Ref-Adv-s. FineCops-Ref reveals a small A4 deficit of 0.52 percentage points at [email protected] (95% CI [0.12, 0.95] in favor of S4), but A8 recovers it and finishes 0.26 points above S4. Official FineCops levels do not show a monotonic increase in the gap. A4 reduces trainable decoder parameters by 44.4% and cached-decoder latency by 10.1%, although end-to-end latency remains backbone-dominated. These results concern the trainable grounding decoder, not a complete attention-only VLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 11 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

11 of 11 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  5. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 240
  6. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 120
  7. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 83
  8. 2024F-LMM: Grounding Frozen Large Multimodal Modelscited by 34
  9. 2024FineCops-Ref: A new Dataset and Task for Fine-Grained Compositional Referring Expression Comprehensioncited by 20
  10. 2022When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarkscited by 11
  11. 2026A Controlled Study of Attention-Only Transformerscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
  2. 2022Improving Visual Grounding by Encouraging Consistent Gradient-based Explanations
  3. 2026Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models
  4. 2026Large Vision-Language Models Get Lost in Attention
  5. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects
  6. 2026Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models
  7. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancement
  8. 2026From Patching to Pruning Visual Computation in Vision Language Models