mlsanityData to 7 Oct 2026

ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling

arXiv 2402.06118 ↗First posted 9 Feb 2024cs.CV cs.AIcited by 34

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

By combining natural language understanding, generation capabilities, and breadth of knowledge of large language models with image perception, recent large vision language models (LVLMs) have shown unprecedented visual reasoning capabilities. However, the generated text often suffers from inaccurate grounding in the visual input, resulting in errors such as hallucination of nonexistent scene elements, missing significant parts of the scene, and inferring incorrect attributes of and relationships between objects. To address these issues, we introduce a novel framework, ViGoR (Visual Grounding Through Fine-Grained Reward Modeling) that utilizes fine-grained reward modeling to significantly enhance the visual grounding of LVLMs over pre-trained baselines. This improvement is efficiently achieved using much cheaper human evaluations instead of full supervisions, as well as automated methods. We show the effectiveness of our approach through a variety of evaluation methods and benchmarks. Additionally, we released our human annotation (https://github.com/amazon-science/vigor) comprising 15,440 images and generated text pairs with fine-grained evaluations to contribute to related research in the community.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 50 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

42 of 50 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
30 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2021Grounded Language-Image Pre-trainingcited by 1,147
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  14. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  15. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  16. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  17. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  18. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  19. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  20. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  21. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  22. 2023PACO: Parts and Attributes of Common Objectscited by 146
  23. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 76
  24. 2022Implicit Autoencoder for Point-Cloud Self-Supervised Representation Learningcited by 46
  25. 2021HPNet: Deep Primitive Segmentation Using Hybrid Representationscited by 35
  26. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistantscited by 25
  27. 2019Extreme Relative Pose Network under Hybrid Representationscited by 19
  28. 20233D Feature Prediction for Masked-AutoEncoder-Based Point Cloud Pretrainingcited by 13
  29. 2019Recurrent Feedback Improves Feedforward Representations in Deep Neural Networkscited by 5
  30. 2023Multi-View Representation is What You Need for Point-Cloud Pre-Trainingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 17; 2026: 3.

34 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  2. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 98
  3. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  4. 2024Toward Visual Grounding: A Surveycited by 52
  5. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  6. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  7. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  8. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  9. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  10. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  11. 2024The Evolution of Multimodal Model Architecturescited by 21
  12. 2024Instruction-Guided Visual Maskingcited by 17
22 more
  1. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  2. 2025OwlCap: Harmonizing Motion-Detail for Video Captioning via HMD-270K and Caption Set Equivalence Rewardcited by 11
  3. 2024EACO: Enhancing Alignment in Multimodal LLMs via Critical Observationcited by 8
  4. 2025VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignmentcited by 8
  5. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 7
  6. 2024Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysiscited by 4
  7. 2025A Structured Review of Underwater Object Detection Challenges and Solutions: From Traditional to Large Vision Language Modelscited by 4
  8. 2025TRACE: Textual Relevance Augmentation and Contextual Encoding for Multimodal Hate Detectioncited by 3
  9. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucinationcited by 2
  10. 2025Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensemblescited by 2
  11. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  12. 2025Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Imagescited by 1
  13. 2025MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoningcited by 1
  14. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  15. 2025IRIS: Intrinsic Reward Image Synthesiscited by 1
  16. 2025VaPR -- Vision-language Preference alignment for Reasoningcited by 1
  17. 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
  18. 2025Token-Level Inference-Time Alignment for Vision-Language Modelscited by 0
  19. 2025Self-Supervised Masked Autoencoders with Dense-Unet for Coronary Calcium Removal in limited CT Datacited by 0
  20. 2026SpatialPoint: Spatial-aware Point Prediction for Embodied Localizationcited by 0
  21. 2026Progressive Reasoning with Primitive Correction for Compositional Zero-Shot Learningcited by 0
  22. 2026Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models
  2. 2026ViGoR-Bench: How Far Are Visual Generative Models From Zero-Shot Visual Reasoners?
  3. 2025Towards Understanding Visual Grounding in Visual Language Models
  4. 2025Grounded Reinforcement Learning for Visual Reasoning
  5. 2026GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models
  6. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
  7. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  8. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding