mlsanityData to 14 Oct 2026

Generating Faithful and Salient Text from Multimodal Data

arXiv 2409.03961 ↗First posted 6 Sep 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

While large multimodal models (LMMs) have obtained strong performance on many multimodal tasks, they may still hallucinate while generating text. Their performance on detecting salient features from visual data is also unclear. In this paper, we develop a framework to generate faithful and salient text from mixed-modal data, which includes images and structured data ( represented in knowledge graphs or tables). Specifically, we train a small vision critic model to identify hallucinated and non-salient features from the image modality. The critic model also generates a list of salient image features. This information is used in the post editing step to improve the generation quality. Experiments on two datasets show that our framework improves LMMs' generation quality on both faithfulness and saliency, outperforming recent techniques aimed at reducing hallucination.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 43 references that are in the map. First posted on arXiv 6 Sep 2024.

Builds on

29 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  2. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 27

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2015VQA: Visual Question Answeringcited by 2,506
15 more
  1. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  5. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  6. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  7. 2019Long and Diverse Text Generation with Planning-based Hierarchical Variational Modelcited by 62
  8. 2022GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generationcited by 16
  9. 2022PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generationcited by 16
  10. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  11. 2024Unifying Structured Data as Graph for Data-to-Text Pre-Trainingcited by 13
  12. 2023Localized Symbolic Knowledge Distillation for Visual Commonsense Modelscited by 12
  13. 2022Self-supervised Graph Masking Pre-training for Graph-to-Text Generationcited by 8
  14. 2020Boosting House Price Predictions using Geo-Spatial Network Embeddingcited by 7
  15. 2023Multimodal Prompt Learning for Product Title Generation with Extremely Limited Labelscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024MEGL: Multimodal Explanation-Guided Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
  2. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding
  3. 2025A Survey of Generative Categories and Techniques in Multimodal Generative Models
  4. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models
  5. 2025A Survey of Multimodal Hallucination Evaluation and Detection
  6. 2024Benchmarking Large Multimodal Models against Common Corruptions
  7. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  8. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training