mlsanityData to 7 Oct 2026

GiVE: Guiding Visual Encoder to Perceive Overlooked Information

arXiv 2410.20109 ↗First posted 26 Oct 2024cs.CV cs.AI cs.MMcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models have advanced AI in applications like text-to-video generation and visual question answering. These models rely on visual encoders to convert non-text data into vectors, but current encoders either lack semantic alignment or overlook non-salient objects. We propose the Guiding Visual Encoder to Perceive Overlooked Information (GiVE) approach. GiVE enhances visual representation with an Attention-Guided Adapter (AG-Adapter) module and an Object-focused Visual Semantic Learning module. These incorporate three novel loss terms: Object-focused Image-Text Contrast (OITC) loss, Object-focused Image-Image Contrast (OIIC) loss, and Object-focused Image Discrimination (OID) loss, improving object consideration, retrieval accuracy, and comprehensiveness. Our contributions include dynamic visual focus adjustment, novel loss functions to enhance object retrieval, and the Multi-Object Instruction (MOInst) dataset. Experiments show our approach achieves state-of-the-art performance.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 55 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

39 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference21 Mar
  2. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  4. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  5. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  6. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs17 Jul
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  8. 2023A Survey on Multimodal Large Language Models23 Jun
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
18 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  7. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  8. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  9. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  10. 2023Demystifying CLIP Datacited by 265
  11. 2020Eigen-CAM: Class Activation Map using Principal Componentscited by 188
  12. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  13. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  14. 2021Large-scale Unsupervised Semantic Segmentationcited by 98
  15. 2023SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMscited by 54
  16. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignmentcited by 28
  17. 2023DeepMIM: Deep Supervision for Masked Image Modelingcited by 16
  18. 2023Masked Image Residual Learning for Scaling Deeper Vision Transformerscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  2. 2026AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commercecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  2. 2022REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory
  3. 2025Visual Representations inside the Language Model
  4. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models
  5. 2025How Can Objects Help Video-Language Understanding?
  6. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  7. 2026Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness
  8. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data