mlsanityData to 7 Oct 2026

BRAVE: Broadening the visual encoding of vision-language models

arXiv 2404.07204 ↗First posted 10 Apr 2024cs.CV cs.AI cs.LGcited by 68

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) are typically composed of a vision encoder, e.g. CLIP, and a language model (LM) that interprets the encoded features to solve downstream tasks. Despite remarkable progress, VLMs are subject to several shortcomings due to the limited capabilities of vision encoders, e.g. "blindness" to certain image features, visual hallucination, etc. To address these issues, we study broadening the visual encoding capabilities of VLMs. We first comprehensively benchmark several vision encoders with different inductive biases for solving VLM tasks. We observe that there is no single encoding configuration that consistently achieves top performance across different tasks, and encoders with different biases can perform surprisingly similarly. Motivated by this, we introduce a method, named BRAVE, that consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen LM. BRAVE achieves state-of-the-art performance on a broad range of captioning and VQA benchmarks and significantly reduces the aforementioned issues of VLMs, while requiring a smaller number of trainable parameters than existing methods and having a more compressed representation. Our results highlight the potential of incorporating different visual biases for a more broad and contextualized visual understanding of VLMs.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 87 of its 99 references that are in the map. First posted on arXiv 10 Apr 2024.

Builds on

87 of 99 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Pixel Aligned Language Models14 Dec
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 81

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  13. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  14. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  15. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  16. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  17. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  18. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  19. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  20. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  21. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2021Scaling Vision Transformerscited by 993
  24. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  25. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  26. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  27. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  28. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  29. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  30. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  31. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  32. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  33. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  34. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  35. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  36. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  37. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  38. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 34; 2026: 16.

68 papers.

Sort:

By area:

  1. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  2. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  3. 2024PaliGemma 2: A Family of Versatile VLMs for Transfercited by 204
  4. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  5. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  6. 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
  7. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  8. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  9. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  10. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  11. 2025FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performancecited by 31
  12. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
38 more
  1. 2024Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusioncited by 18
  2. 2024Law of Vision Representation in MLLMscited by 16
  3. 2025GenHancer: Imperfect Generative Models are Secretly Strong Vision-Centric Enhancerscited by 16
  4. 2024Phantom of Latent for Large Language and Vision Modelscited by 15
  5. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  6. 2025CompoDistill: Attention Distillation for Compositional Reasoning in Multimodal LLMscited by 13
  7. 2025Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skippingcited by 11
  8. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  9. 2025Unifying Specialized Visual Encoders for Video Language Modelscited by 10
  10. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  11. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMscited by 7
  12. 2025Connector-S: A Survey of Connectors in Multi-modal Large Language Modelscited by 7
  13. 2025Building Trustworthy Multimodal AI: A Review of Fairness, Transparency, and Ethics in Vision-Language Taskscited by 7
  14. 2024Learning Visual Prompts for Guiding the Attention of Vision Transformerscited by 6
  15. 2024Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learningcited by 6
  16. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniquescited by 5
  17. 2025Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoderscited by 5
  18. 2025NAUTILUS: A Large Multimodal Model for Underwater Scene Understandingcited by 5
  19. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
  20. 2025TokenFLEX: Unified VLM Training for Flexible Visual Tokens Inferencecited by 4
  21. 2025LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Expertscited by 4
  22. 2025FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representationscited by 4
  23. 2025Exploring The Visual Feature Space for Multimodal Neural Decodingcited by 4
  24. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 4
  25. 2025Multi-Agent Interactive Question Generation Framework for Long Document Understandingcited by 4
  26. 2025HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Modelscited by 4
  27. 2026OmniStream: Mastering Perception, Reconstruction and Action in Continuous Streamscited by 4
  28. 2025Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractorcited by 3
  29. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  30. 2025RL makes MLLMs see better than SFTcited by 3
  31. 2025Sparse Autoencoders are Topic Modelscited by 3
  32. 2024Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledgecited by 2
  33. 2025Diffusion Instruction Tuningcited by 2
  34. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?cited by 2
  35. 2025un2CLIP: Improving CLIP's Visual Detail Capturing Ability via Inverting unCLIPcited by 2
  36. 2025Q-Former Autoencoder: A Modern Framework for Medical Anomaly Detectioncited by 2
  37. 2025Scope: Selective Cross-modal Orchestration of Visual Perception Expertscited by 2
  38. 2025On the Domain Robustness of Contrastive Vision-Language Modelscited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Imperfect Vision Encoders: Efficient and Robust Tuning for Vision-Language Models
  2. 2024Unveiling Encoder-Free Vision-Language Models
  3. 2026Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
  4. 2025Hidden in plain sight: VLMs overlook their visual representations
  5. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
  6. 2025LLMs Can Compensate for Deficiencies in Visual Representations
  7. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
  8. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilities