OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Large vision-language models (VLMs) often struggle to generate long and factual captions. However, traditional measures for hallucination and factuality are not well suited for evaluating longer, more diverse captions and in settings where ground-truth human-annotated captions are unavailable. We introduce OV-Fact, a novel method for measuring caption factuality of long captions that leverages open-vocabulary visual grounding and tool-based verification without depending on human annotations. Our method improves agreement with human judgments and captures both caption descriptiveness (recall) and factual precision in the same metric. Furthermore, unlike previous metrics, our reference-free method design enables new applications towards factuality-based data filtering. We observe models trained on an OVFact-filtered (2.5-5x less) subset of a large-scale, noisy (VLM-generated) pretraining set meaningfully improve factuality precision without sacrificing caption descriptiveness across a range of downstream long caption benchmarks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 63 references that are in the map. First posted on arXiv 25 Jul 2025.
Builds on
42 of 63 references.
Closest ancestors · 4
Same direction, newest first
- 2024PaliGemma 2: A Family of Versatile VLMs for Transfer4 Dec
- 2024BLIP3-KALE: Knowledge Augmented Large-Scale Dense Captions12 Nov
- 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
- 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
Other references · 38
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2018Datasheets for Datasetscited by 1,386
26 more
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
- 2018Model Cards for Model Reportingcited by 1,080
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
- 2018Panoptic Segmentationcited by 802
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
- 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
- 2023DataComp: In search of the next generation of multimodal datasetscited by 644
- 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
- 2019Connecting Vision and Language with Localized Narrativescited by 226
- 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
- 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
- 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 107
- 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
- 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
- 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
- 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
- 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
- 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 45
- 2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
- 2024Data curation via joint example selection further accelerates multimodal learningcited by 33
- 2024Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generationcited by 28
- 2024Active Data Curation Effectively Distills Large-Scale Multimodal Modelscited by 19
- 2024SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Imagescited by 5
- 2023The Role of Data Curation in Image Captioningcited by 4
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Unblocking Fine-Grained Evaluation of Detailed Captions: An Explaining AutoRater and Critic-and-Revise Pipeline
- 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage
- 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
- 2025VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
- 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
- 2024Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions
- 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis
- 2025Towards Statistical Factuality Guarantee for Large Vision-Language Models