# How to Detect and Defeat Molecular Mirage: A Metric-Driven Benchmark for Hallucination in LLM-based Molecular Comprehension

arXiv: https://arxiv.org/abs/2504.12314 · HTML: https://mlsanity.com/paper/2504.12314/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 69 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 30 of 38)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021)
- [A Survey of Hallucination in Large Foundation Models](https://mlsanity.com/paper/2309.05922/) (2023)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [Pre-training Molecular Graph Representation with 3D Geometry](https://mlsanity.com/paper/2110.07728/) (2021)
- [Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?](https://mlsanity.com/paper/2405.05904/) (2024)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples](https://mlsanity.com/paper/2310.01469/) (2023)
- [Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing](https://mlsanity.com/paper/2212.10789/) (2022)
- [A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language](https://mlsanity.com/paper/2209.05481/) (2022)
- [MolCA: Molecular Graph-Language Modeling with Cross-Modal Projector and Uni-Modal Adapter](https://mlsanity.com/paper/2310.12798/) (2023)
- [LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset](https://mlsanity.com/paper/2402.09391/) (2024)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019)
- [Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding](https://mlsanity.com/paper/2104.08455/) (2021)
- [BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicine](https://mlsanity.com/paper/2308.09442/) (2023)
- [BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations](https://mlsanity.com/paper/2310.07276/) (2023)
- [InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery](https://mlsanity.com/paper/2311.16208/) (2023)
- [Scientific Large Language Models: A Survey on Biological & Chemical Domains](https://mlsanity.com/paper/2401.14656/) (2024)
- [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://mlsanity.com/paper/2403.17752/) (2024)
- [Keeping it Simple: Language Models can learn Complex Molecular Distributions](https://mlsanity.com/paper/2112.03041/) (2021)
- [Towards 3D Molecule-Text Interpretation in Language Models](https://mlsanity.com/paper/2401.13923/) (2024)
- [EvaGaussians: Event Stream Assisted Gaussian Splatting from Blurry Images](https://mlsanity.com/paper/2405.20224/) (2024)
- [ProLLaMA: A Protein Large Language Model for Multi-Task Protein Language Processing](https://mlsanity.com/paper/2402.16445/) (2024)
- [Fake Alignment: Are LLMs Really Aligned Well?](https://mlsanity.com/paper/2311.05915/) (2023)

## Built on by (most cited first, 6 of 6)

- [Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metrics](https://mlsanity.com/paper/2505.18658/) (2025)
- [LatentChem: From Textual CoT to Latent Thinking in Chemical Reasoning](https://mlsanity.com/paper/2602.07075/) (2026)
- [Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesis](https://mlsanity.com/paper/2601.17687/) (2026)
- [Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?](https://mlsanity.com/paper/2506.10912/) (2025)
- [MoleCode unlocks structural intelligence in large language models](https://mlsanity.com/paper/2605.16480/) (2026)
- [False Fixed Points: Kantian Feedback, Stable Miscalibration, and Representational Compression in LLMs](https://mlsanity.com/paper/2510.14925/) (2025)

## Abstract (arXiv)

Large language models are increasingly used in scientific domains, especially for molecular understanding and analysis. However, existing models are affected by hallucination issues, resulting in errors in drug design and utilization. In this paper, we first analyze the sources of hallucination in LLMs for molecular comprehension tasks, specifically the knowledge shortcut phenomenon observed in the PubChem dataset. To evaluate hallucination in molecular comprehension tasks with computational efficiency, we introduce Mol-Hallu, a novel free-form evaluation metric that quantifies the degree of hallucination based on the scientific entailment relationship between generated text and actual molecular properties. Utilizing the Mol-Hallu metric, we reassess and analyze the extent of hallucination in various LLMs performing molecular comprehension tasks. Furthermore, the Hallucination Reduction Post-processing stage~(HRPP) is proposed to alleviate molecular hallucinations, Experiments show the effectiveness of HRPP on decoder-only and encoder-decoder molecular LLMs. Our findings provide critical insights into mitigating hallucination and improving the reliability of LLMs in scientific applications.