# Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation

arXiv: https://arxiv.org/abs/2404.19752 · HTML: https://mlsanity.com/paper/2404.19752/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 41 references that are in the map.
- First posted on arXiv 30 Apr 2024.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework](https://mlsanity.com/paper/2202.03052/) (2022)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Detecting and Preventing Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2308.06394/) (2023)
- [Scalable 3D Captioning with Pretrained Models](https://mlsanity.com/paper/2306.07279/) (2023)
- [PromptCap: Prompt-Guided Task-Aware Image Captioning](https://mlsanity.com/paper/2211.09699/) (2022)
- [Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training](https://mlsanity.com/paper/2210.08773/) (2022)
- [ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions](https://mlsanity.com/paper/2303.06594/) (2023)
- [VIGC: Visual Instruction Generation and Correction](https://mlsanity.com/paper/2308.12714/) (2023)
- [Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning](https://mlsanity.com/paper/2206.01843/) (2022)

## Built on by (most cited first, 28 of 28)

- [Structured 3D Latents for Scalable and Versatile 3D Generation](https://mlsanity.com/paper/2412.01506/) (2024)
- [DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception](https://mlsanity.com/paper/2407.08303/) (2024)
- [Scaling Vision Pre-Training to 4K Resolution](https://mlsanity.com/paper/2503.19903/) (2025)
- [Learning Visual Grounding from Generative Vision and Language Model](https://mlsanity.com/paper/2407.14563/) (2024)
- [Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage](https://mlsanity.com/paper/2412.15484/) (2024)
- [Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks](https://mlsanity.com/paper/2501.08326/) (2025)
- [Perception in Reflection](https://mlsanity.com/paper/2504.07165/) (2025)
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation](https://mlsanity.com/paper/2506.08210/) (2025)
- [From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment](https://mlsanity.com/paper/2406.13912/) (2024)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioning](https://mlsanity.com/paper/2507.12841/) (2025)
- [TeRA: Rethinking Text-guided Realistic 3D Avatar Generation](https://mlsanity.com/paper/2509.02466/) (2025)
- [ExCap3D: Expressive 3D Scene Understanding via Object Captioning with Varying Detail](https://mlsanity.com/paper/2503.17044/) (2025)
- [Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors](https://mlsanity.com/paper/2509.21997/) (2025)
- [OVFact: Measuring and Improving Open-Vocabulary Factuality for Long Caption Models](https://mlsanity.com/paper/2507.19262/) (2025)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captions](https://mlsanity.com/paper/2505.22627/) (2025)
- [Temporal Visual Semantics-Induced Human Motion Understanding with Large Language Models](https://mlsanity.com/paper/2512.22249/) (2025)
- [One-shot Optimized Steering Vector for Hallucination Mitigation for VLMs](https://mlsanity.com/paper/2601.23041/) (2026)
- [Visual Reasoning Agent: Robust Vision Systems in Remote Sensing via Inference-Time Scaling](https://mlsanity.com/paper/2509.16343/) (2025)
- [Generating Accurate and Detailed Captions for High-Resolution Images](https://mlsanity.com/paper/2510.27164/) (2025)
- [SweeperBot: Making 3D Browsing Accessible through View Analysis and Visual Question Answering](https://mlsanity.com/paper/2511.14567/) (2025)
- [SesaHand: Enhancing 3D Hand Reconstruction via Controllable Generation with Semantic and Structural Alignment](https://mlsanity.com/paper/2603.00443/) (2026)
- [Holo-Captioning: Toward the Text Equivalent of 3D Scenes](https://mlsanity.com/paper/2607.02908/) (2026)
- [A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions](https://mlsanity.com/paper/2607.23235/) (2026)
- [A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss](https://mlsanity.com/paper/2609.00591/) (2026)

## Abstract (arXiv)

Existing automatic captioning methods for visual content face challenges such as lack of detail, content hallucination, and poor instruction following. In this work, we propose VisualFactChecker (VFC), a flexible training-free pipeline that generates high-fidelity and detailed captions for both 2D images and 3D objects. VFC consists of three steps: 1) proposal, where image-to-text captioning models propose multiple initial captions; 2) verification, where a large language model (LLM) utilizes tools such as object detection and VQA models to fact-check proposed captions; 3) captioning, where an LLM generates the final caption by summarizing caption proposals and the fact check verification results. In this step, VFC can flexibly generate captions in various styles following complex instructions. We conduct comprehensive captioning evaluations using four metrics: 1) CLIP-Score for image-text similarity; 2) CLIP-Image-Score for measuring the image-image similarity between the original and the reconstructed image generated by a text-to-image model using the caption. 3) human study on Amazon Mechanical Turk; 4) GPT-4V for fine-grained evaluation. Evaluation results show that VFC outperforms state-of-the-art open-sourced captioning methods for 2D images on the COCO dataset and 3D assets on the Objaverse dataset. Our study demonstrates that by combining open-source models into a pipeline, we can attain captioning capability comparable to proprietary models such as GPT-4V, despite being over 10x smaller in model size.