mlsanityData to 7 Oct 2026

Hallucination Detection for Grounded Instruction Generation

arXiv 2310.15319 ↗First posted 23 Oct 2023cs.CL cs.AI cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We investigate the problem of generating instructions to guide humans to navigate in simulated residential environments. A major issue with current models is hallucination: they generate references to actions or objects that are inconsistent with what a human follower would perform or encounter along the described path. We develop a model that detects these hallucinated references by adopting a model pre-trained on a large corpus of image-text pairs, and fine-tuning it with a contrastive loss that separates correct instructions from instructions containing synthesized hallucinations. Our final model outperforms several baselines, including using word probability estimated by the instruction-generation model, and supervised models based on LSTM and Transformer.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

37 of 50 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  10. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  11. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  12. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
25 more
  1. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  2. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  3. 2017Challenges in Data-to-Document Generationcited by 226
  4. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  5. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  6. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 151
  7. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  8. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  9. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  10. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  11. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  12. 2022Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Bettercited by 75
  13. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  14. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  15. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  16. 2019Domain Robustness in Neural Machine Translationcited by 64
  17. 2021Less is More: Generating Grounded Navigation Instructions from Landmarkscited by 62
  18. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  19. 2017Unified Pragmatic Models for Generating and Following Instructionscited by 55
  20. 2021Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuningcited by 46
  21. 2021On the Evaluation of Vision-and-Language Navigation Instructionscited by 45
  22. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  23. 2021Controlling Hallucinations at Word Level in Data-to-Text Generationcited by 41
  24. 2019Multi-modal Discriminative Model for Vision-and-Language Navigationcited by 17
  25. 2022Define, Evaluate, and Improve Task-Oriented Cognitive Capabilities for Instruction Generation Modelscited by 7

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 2.

7 papers.

  1. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  2. 2024DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Modelscited by 15
  3. 2025Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detectioncited by 8
  4. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 7
  5. 2024BSharedRAG: Backbone Shared Retrieval-Augmented Generation for the E-commerce Domaincited by 4
  6. 2024Successfully Guiding Humans with Imperfect Instructions by Highlighting Potential Errors and Suggesting Correctionscited by 2
  7. 2025ExpertGenQA: Open-ended QA generation in Specialized Domainscited by 0