mlsanityData to 7 Oct 2026

Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering

arXiv 2605.01827 ↗First posted 3 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) have recently demonstrated their proficiency in holistic visual comprehension. However, most of them struggle to tackle region-level perception guided by visual prompts, especially for cases where multiple regions are referred simultaneously, or scenarios where global contexts are necessary for precise visual referring. We introduce Contextual Latent Steering (CSteer), a training-free approach for guiding general LMMs to refer multiple regions contextually, without expensive fine-tuning or architectural modifications. CSteer starts with pre-computing contextual vectors that implicitly represent visual referring behaviors, such as differentiation among regions and attention to global contexts, followed by representation editing during inference time. Experimental results on multiple datasets indicate that general LMMs with CSteer outperform tailored referring LMMs in most cases, suggesting a promising solution in training-free, and setting new state-of-the-art for this field. Code is available at https://github.com/xing0047/csteer.git.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 71 of its 79 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

71 of 79 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025PixelRefer: A Unified Framework for Spatio-Temporal Object Referring with Arbitrary Granularity27 Oct
  2. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs21 Oct
  3. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs24 Feb
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  8. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want29 Mar
  9. 2023Osprey: Pixel Understanding with Visual Instruction Tuning15 Dec
  10. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
2 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 57

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
38 more
  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  3. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  4. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  5. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  6. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  7. 2025Seed1.5-VL Technical Reportcited by 367
  8. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  9. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  10. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  11. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  12. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  13. 2023Teaching CLIP to Count to Tencited by 194
  14. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  15. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  16. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  17. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  18. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  19. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  20. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  21. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  22. 2024Large Multimodal Agents: A Surveycited by 99
  23. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  24. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  25. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
  26. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  27. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 71
  28. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  29. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  30. 2024Multi-Object Hallucination in Vision-Language Modelscited by 60
  31. 2024CountGD: Multi-Modal Open-World Countingcited by 59
  32. 2025MOSEv2: A More Challenging Dataset for Video Object Segmentation in Complex Scenescited by 59
  33. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 54
  34. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  35. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 40
  36. 2020Point and Ask: Incorporating Pointing into Visual Question Answeringcited by 39
  37. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  38. 2024EarthMarker: A Visual Prompting Multi-modal Large Language Model for Remote Sensingcited by 29

50 most cited of 57 All 57 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
  2. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Models
  3. 2026Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception
  4. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
  5. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  6. 2025Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models
  7. 2025Multimodal Function Vectors for Visual Relations
  8. 2025Region-Level Context-Aware Multimodal Understanding