mlsanityData to 13 Oct 2026

Vision as LoRA

arXiv 2503.20680 ↗First posted 26 Mar 2025cs.CV cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

We introduce Vision as LoRA (VoRA), a novel paradigm for transforming an LLM into an MLLM. Unlike prevalent MLLM architectures that rely on external vision modules for vision encoding, VoRA internalizes visual capabilities by integrating vision-specific LoRA layers directly into the LLM. This design allows the added parameters to be seamlessly merged into the LLM during inference, eliminating structural complexity and minimizing computational overhead. Moreover, inheriting the LLM's ability of handling flexible context, VoRA can process inputs at arbitrary resolutions. To further strengthen VoRA's visual capabilities, we introduce a block-wise distillation method that transfers visual priors from a pre-trained ViT into the LoRA layers, effectively accelerating training by injecting visual knowledge. Additionally, we apply bi-directional attention masks to better capture the context information of an image. We successfully demonstrate that with additional pre-training data, VoRA can perform comparably with conventional encode-based MLLMs. All training data, codes, and model weights will be released at https://github.com/Hon-Wong/VoRA.

Key facts. Cited by 37 papers in the map (data through 13 Oct 2026). Builds on 47 of its 53 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

47 of 53 references.

References · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,180
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  7. 2023Visual Instruction Tuningcited by 7,827
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  9. 2024Qwen2.5 Technical Reportcited by 4,840
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
35 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  2. 2023Qwen Technical Reportcited by 3,310
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,692
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,118
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,896
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  12. 2019Towards VQA Models That Can Readcited by 1,705
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
  14. 2019Root Mean Square Layer Normalizationcited by 1,449
  15. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,230
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,041
  17. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 827
  18. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 731
  19. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 706
  20. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 689
  21. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 654
  22. 2023DataComp: In search of the next generation of multimodal datasetscited by 639
  23. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 621
  24. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 484
  25. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 475
  26. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 308
  27. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 243
  28. 2024Pixtral 12Bcited by 177
  29. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  30. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 109
  31. 2024Unveiling Encoder-Free Vision-Language Modelscited by 92
  32. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  33. 2024Elysium: Exploring Object-level Perception in Videos via MLLMcited by 48
  34. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 39
  35. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 11; 2026: 26.

37 papers.

Sort:

By area:

  1. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  2. 2025Prolonged Reasoning Is Not All You Need: Certainty-Based Adaptive Routing for Efficient LLM/MLLM Reasoningcited by 34
  3. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 28
  4. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  5. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 16
  6. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  7. 2026Xiaomi OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanationcited by 9
  8. 2025Context-Aware Autoregressive Models for Multi-Conditional Image Generationcited by 5
  9. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraintscited by 5
  10. 2025ChineseVideoBench: Benchmarking Multi-modal Large Models for Chinese Video Question Answeringcited by 4
  11. 2025SigLino: Efficient Multi-Teacher Distillation for Agglomerative Vision Foundation Modelscited by 2
  12. 2026A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluationcited by 2
25 more
  1. 2026A Unified Language Model for Large Scale Search, Recommendation, and Reasoningcited by 2
  2. 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMscited by 1
  3. 2026The Override Gap: A Magnitude Account of Knowledge Conflict Failure in Hypernetwork-Based Instant LLM Adaptationcited by 1
  4. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  5. 2025Inverse-LLaVA: Rethinking Multimodal Alignment via Text-to-Vision Mappingcited by 0
  6. 2025Selective LoRA for Visual Tokens and Attention Headscited by 0
  7. 2026LPCAN: Lightweight Pyramid Cross-Attention Network for Rail Surface Defect Detection Using RGB-D Datacited by 0
  8. 2026Knowledge-Embedded and Hypernetwork-Guided Few-Shot Substation Meter Defect Image Generation Methodcited by 0
  9. 2026MA-LipNet: Multi-Dimensional Attention Networks for Robust Lipreadingcited by 0
  10. 2026The Diffusion Duet: Harmonizing Dual Channels with Wavelet Suppression for Image Separationcited by 0
  11. 2026Detecting Deepfakes with Multivariate Soft Blending and CLIP-based Image-Text Alignmentcited by 0
  12. 2026Efficient Image Super-Resolution with Multi-Scale Spatial Adaptive Attention Networkscited by 0
  13. 2026A Deformable Attention-Based Detection Transformer with Cross-Scale Feature Fusion for Industrial Coil Spring Inspectioncited by 0
  14. 2026Transmittance-Guided Structure-Texture Decomposition for Nighttime Image Dehazingcited by 0
  15. 2026Hierarchical Awareness Adapters with Hybrid Pyramid Feature Fusion for Dense Depth Predictioncited by 0
  16. 2026Image Classification via Random Dilated Convolution with Multi-Branch Feature Extraction and Context Excitationcited by 0
  17. 2026Multi-Branch Non-Homogeneous Image Dehazing via Concentration Partitioning and Image Fusioncited by 0
  18. 2026Local Spatiotemporal Convolutional Network for Robust Gait Recognitioncited by 0
  19. 2026Do You Need Text Rectification? Soft Attention Mask Embedding for Rectification-Free Scene Text Spottingcited by 0
  20. 2026The Hidden Evolution of Disguised Visual Context inside the VLMcited by 0
  21. 2026DAIN: Dynamic Agent-Based Interaction Network for Efficient and Collaborative Multimodal Reasoningcited by 0
  22. 2026ProWAFT: A ROMA-LPD Instance for Workload-Aware and Dynamic Fault Tolerance in FPGA-Based CNN Acceleratorscited by 0
  23. 2026CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioningcited by 0
  24. 2026Rank-Consistent Set Reasoning for Co-Salient Object Detectioncited by 0
  25. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LangVision-LoRA-NAS: Neural Architecture Search for Variable LoRA Rank in Vision Language Models
  2. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models
  3. 2024Empower Vision Applications with LoRA LMM
  4. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Models
  5. 2024Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference
  6. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models
  7. 2026Youtu-VL: Unleashing Visual Potential via Unified Vision-Language Supervision
  8. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks