Timage: A Generative Text-in-Image Paradigm for Fine-Tuning Vision-Language Models
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) often lose track of the right image regions during fine-grained spatial reasoning, because a textual query rarely carries any explicit geometric anchor into the pixel domain. Prevailing remedies either rewire the model's weights or pad the prompt with verbose instructions, yet neither reliably pins the language to the correct visual coordinates without eroding the backbone's general competence. We introduce Timage, a paradigm that recasts multimodal understanding as an alignment problem solved at the input: the query is drawn, as a typeset overlay, onto the image itself. The placement and appearance of this overlay are produced by a Constrained Schr"odinger Bridge (cSB), an entropic optimal-transport sampler that factorizes layout synthesis into two coupled stochastic stages. The first stage, Region Search, transports noise toward query-aligned image zones while obeying a hard occlusion barrier that protects salient foreground content; the second stage, Appearance Shaping, sizes the glyphs through an ``ink-budget'' regularizer so that the rendered text stays legible and visually balanced. The resulting overlay behaves as an explicit attention beacon that channels the model's focus along spatial semantics. On the VMCBench suite, Timage paired with a modest 7B backbone clearly overtakes far larger proprietary systems as well as parameter-tuned baselines. The study positions deliberate input reconstruction as a powerful, architecture-neutral lever for strengthening multimodal reasoning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 48 of its 57 references that are in the map. First posted on arXiv 18 Jun 2026.
Builds on
48 of 57 references.
References · 48
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Denoising Diffusion Implicit Modelscited by 9,781
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2017Neural Discrete Representation Learningcited by 4,712
- 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
- 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
- 2022Visual Prompt Tuningcited by 1,681
- 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
36 more
- 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
- 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
- 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
- 2016Conditional Image Generation with PixelCNN Decoderscited by 862
- 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
- 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
- 2022Vision Transformer Adapter for Dense Predictionscited by 565
- 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
- 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
- 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
- 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
- 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
- 2022Dual Diffusion Implicit Bridges for Image-to-Image Translationcited by 213
- 2023LCM-LoRA: A Universal Stable-Diffusion Acceleration Modulecited by 211
- 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 198
- 2022Stochastic Trajectory Prediction via Motion Indeterminacy Diffusioncited by 185
- 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 141
- 2023Parallel Sampling of Diffusion Modelscited by 112
- 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 104
- 2022Visual Prompt Tuning for Test-time Domain Adaptationcited by 97
- 2023Unpaired Image-to-Image Translation via Neural Schr"odinger Bridgecited by 96
- 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 96
- 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
- 2017Parallel Multiscale Autoregressive Density Estimationcited by 71
- 2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
- 2023Diversity-Aware Meta Visual Promptingcited by 58
- 2023StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generationcited by 58
- 2024Light and Optimal Schr"odinger Bridge Matchingcited by 45
- 2024MTLoRA: A Low-Rank Adaptation Approach for Efficient Multi-Task Learningcited by 44
- 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluationcited by 30
- 2023Learning Expressive Prompting With Residuals for Vision Transformerscited by 21
- 2024Discrete Diffusion Schr"odinger Bridge Matching for Graph Transformationcited by 21
- 2025Audio Super-Resolution with Latent Bridge Modelscited by 14
- 2025Finding Local Diffusion Schr"odinger Bridge using Kolmogorov-Arnold Networkcited by 8
- 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Modelscited by 4
- 2024Latent Schrodinger Bridge: Prompting Latent Diffusion for Fast Unpaired Image-to-Image Translationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
- 2026SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models
- 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
- 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
- 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
- 2025Visual Representation Alignment for Multimodal Large Language Models
- 2026LookWise: Knowing When and Where to Look for Fine-Grained Visual Reasoning in Multimodal Large Language Models
- 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models