mlsanityData to 7 Oct 2026

Reconstructive Visual Instruction Tuning

arXiv 2410.09575 ↗First posted 12 Oct 2024cs.CV cs.AI cs.CLcited by 54

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 78 of its 105 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

78 of 105 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  5. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  8. 2023Generative Multimodal Models are In-Context Learners20 Dec
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
2 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2024The Llama 3 Herd of Modelscited by 14,192
  10. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  11. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  12. 2023Segment Anythingcited by 8,158
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  13. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  14. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  15. 2024Qwen2 Technical Reportcited by 2,076
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  17. 2019Towards VQA Models That Can Readcited by 1,696
  18. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  19. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  20. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  21. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  22. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  23. 2016A Diagram Is Worth A Dozen Imagescited by 793
  24. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  25. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  26. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  27. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  28. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  29. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  30. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  31. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  32. 2022DeiT III: Revenge of the ViTcited by 417
  33. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  34. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  35. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  36. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  37. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  38. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 23.

54 papers.

Sort:

By area:

  1. 2025DriveVLA-W0: World Models Amplify Data Scaling Law in Autonomous Drivingcited by 127
  2. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  3. 2025VGR: Visual Grounded Reasoningcited by 70
  4. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 67
  5. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  6. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  7. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  8. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  9. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  10. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformercited by 28
  11. 2025Sketch-in-Latents: Eliciting Unified Reasoning in MLLMscited by 24
  12. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
38 more
  1. 2025Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimationcited by 20
  2. 2025SRUM: Fine-Grained Self-Rewarding for Unified Multimodal Modelscited by 19
  3. 2026Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generationcited by 19
  4. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  5. 2025Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Groundingcited by 17
  6. 2025Autoregressive Semantic Visual Reconstruction Helps VLMs Understand Bettercited by 17
  7. 2025GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancerscited by 16
  8. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  9. 2025VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximizationcited by 12
  10. 2025Hita: Holistic Tokenizer for Autoregressive Image Generationcited by 11
  11. 2026ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Modelscited by 11
  12. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 9
  13. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialoguescited by 7
  14. 2026Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuningcited by 7
  15. 2026VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verificationcited by 6
  16. 2025Visual Instruction Bottleneck Tuningcited by 5
  17. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  18. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignmentcited by 5
  19. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  20. 2025Vision-Centric Activation and Coordination for Multimodal Large Language Modelscited by 3
  21. 2025UniHetero: Could Generation Enhance Understanding for Vision-Language-Model at Large Data Scale?cited by 3
  22. 2026Steering Visual Generation in Unified Multimodal Models with Understanding Supervisioncited by 3
  23. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIPcited by 2
  24. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  25. 2026Guiding Diffusion-based Reconstruction with Contrastive Signals for Balanced Visual Representationcited by 2
  26. 2026HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizerscited by 2
  27. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 1
  28. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 1
  29. 2026Boosting Visual Instruction Tuning with Self-Supervised Guidancecited by 1
  30. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 1
  31. 2026Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPOcited by 1
  32. 2026Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Predictioncited by 1
  33. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Modelscited by 0
  34. 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Modelscited by 0
  35. 2026Semantic Generative Tuning for Unified Multimodal Modelscited by 0
  36. 2026S2COPE: Self-Supervised Concept Discovery via Preference Learningcited by 0
  37. 2026Mind the Heads: Topological Representation Alignment for Multimodal LLMscited by 0
  38. 2026StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuningcited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to Instruct for Visual Instruction Tuning
  2. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  3. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  4. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  5. 2025Do we Really Need Visual Instructions? Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
  6. 2024Instruction-Guided Visual Masking
  7. 2023InstructAny2Pix: Flexible Visual Editing via Multimodal Instruction Following
  8. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descent