mlsanityData to 7 Oct 2026

TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning

arXiv 2410.10491 ↗First posted 14 Oct 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Spatial awareness is key to enable embodied multimodal AI systems. Yet, without vast amounts of spatial supervision, current Multimodal Large Language Models (MLLMs) struggle at this task. In this paper, we introduce TWIST & SCOUT, a framework that equips pre-trained MLLMs with visual grounding ability without forgetting their existing image and language understanding skills. To this end, we propose TWIST, a twin-expert stepwise tuning module that modifies the decoder of the language model using one frozen module pre-trained on image understanding tasks and another learnable one for visual grounding tasks. This allows the MLLM to retain previously learned knowledge and skills, while acquiring what is missing. To fine-tune the model effectively, we generate a high-quality synthetic dataset we call SCOUT, which mimics human reasoning in visual grounding. This dataset provides rich supervision signals, describing a step-by-step multimodal reasoning process, thereby simplifying the task of visual grounding. We evaluate our approach on several standard benchmark datasets, encompassing grounded image captioning, zero-shot localization, and visual grounding tasks. Our method consistently delivers strong performance across all tasks, while retaining the pre-trained image understanding capabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 43 of its 50 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

43 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PIN: Positional Insert Unlocks Object Localisation Abilities in VLMs13 Feb
  2. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  3. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  5. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov

Other references · 38

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
26 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  5. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  11. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  12. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  13. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  14. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  15. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  16. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  17. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  18. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  19. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  20. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  21. 2024Pixtral 12Bcited by 174
  22. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 79
  23. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  24. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 22
  25. 2023AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulationcited by 19
  26. 2024Empowering Large Language Models on Robotic Manipulation with Affordance Promptingcited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  2. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
  3. 2026Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding
  4. 2025Towards Understanding Visual Grounding in Visual Language Models
  5. 2026IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models
  6. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  7. 2026Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding
  8. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning