mlsanityData to 7 Oct 2026

Token Warping Helps MLLMs Look from Nearby Viewpoints

arXiv 2604.02870 ↗First posted 3 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Can warping tokens, rather than pixels, help multimodal large language models (MLLMs) understand how a scene appears from a nearby viewpoint? While MLLMs perform well on visual reasoning, they remain fragile to viewpoint changes, as pixel-wise warping is highly sensitive to small depth errors and often introduces geometric distortions. Drawing on theories of mental imagery that posit part-level structural representations as the basis for human perspective transformation, we examine whether image tokens in ViT-based MLLMs serve as an effective substrate for viewpoint changes. We compare forward and backward warping, finding that backward token warping, which defines a dense grid on the target view and retrieves a corresponding source-view token for each grid point, achieves greater stability and better preserves semantic coherence under viewpoint shifts. Experiments on our proposed ViewBench benchmark demonstrate that token-level warping enables MLLMs to reason reliably from nearby viewpoints, consistently outperforming all baselines including pixel-wise warping approaches, spatially fine-tuned MLLMs, and a generative warping method.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 106 of its 128 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

106 of 128 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training28 Sep
  2. 2025Introducing Visual Perception Token into Multimodal Large Language Model24 Feb
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  5. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr

Other references · 101

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2025Qwen2.5-VL Technical Reportcited by 5,418
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
38 more
  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  3. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  4. 2024Depth Anything V2cited by 1,385
  5. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  6. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  8. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  9. 2025Continuous 3D Perception Model with Persistent Statecited by 527
  10. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  11. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 442
  12. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  13. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  14. 2025Kimi-VL Technical Reportcited by 381
  15. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  16. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  17. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  18. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  19. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  20. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  21. 2022Vision Transformer with Deformable Attentioncited by 293
  22. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  23. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  24. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  25. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  26. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  27. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 189
  28. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  29. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  30. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 154
  31. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  32. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  33. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 146
  34. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 144
  35. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  36. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  37. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  38. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131

50 most cited of 101 All 101 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping
  2. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment
  3. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generation
  4. 2025MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustness
  5. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs
  6. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
  7. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
  8. 2026Do Vision Language Models Need to Process Image Tokens?