mlsanityData to 14 Oct 2026

Understanding Unified VLMs: From Attention Probing to Guidance Tuning for Multimodal Generation

arXiv 2610.15227 ↗First posted 10 Oct 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: MLLM training and visual instruction tuning

Abstract (arXiv)

Unified vision-language models (VLMs) perform both text-to-image (T2I) and image-to-text (I2T) generation with one shared Transformer backbone, yet how the backbone processes the condition in either direction remains poorly understood. We study this question from the perspective of conditional guidance: we probe every attention head to show the activation difference between two branches of classifier-free guidance (CFG) with different conditional guidance. On two unified VLMs of autoregressive and masked-diffusion paradigms, the probing reveals a subset of heads that are more sensitive to the change of condition while others are agnostic to it. This characteristic pattern holds in both T2I and I2T generation and is largely stable across data samples, demonstrating the internal mechanism of unified VLMs. Motivated by this, we propose a new inference-time intervention approach for unified multimodal generation, named textitHeadTuned-CFG, which uses the probed condition-change awareness to select and tune the conditional guidance at the attention-head level. Experiments show consistent benefits in both multimodal generation directions of unified VLMs without additional training, and in particular substantial improvements over strong baselines in T2I generation. These results further support the key finding of our probing analysis: the observed pattern of functional specialization is an intrinsic property of unified VLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 35 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

29 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Interpreting Attention Heads for Image-to-Text Information Flow in Large Vision-Language Models22 Sep
  2. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Classifier-Free Diffusion Guidancecited by 5,847
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  6. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  11. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  12. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
15 more
  1. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  3. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  4. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 256
  5. 2024Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Modelscited by 254
  6. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  7. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  8. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 203
  9. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 154
  10. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  11. 2024Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidancecited by 113
  12. 2024Analysis of Classifier-Free Guidance Weight Schedulerscited by 68
  13. 2024Smoothed Energy Guidance: Guiding Diffusion Models with Reduced Energy Curvature of Attentioncited by 59
  14. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 55
  15. 2025Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
  2. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformers
  3. 2025Query-Kontext: An Unified Multimodal Model for Image Generation and Editing
  4. 2026CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models
  5. 2025UniFusion: Vision-Language Model as Unified Encoder in Image Generation
  6. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  7. 2026Where Does Vision Meet Language? Understanding and Refining Visual Fusion in MLLMs via Contrastive Attention
  8. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs