# Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs

arXiv: https://arxiv.org/abs/2305.12191 · HTML: https://mlsanity.com/paper/2305.12191/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 30 of 31)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [FaithDial: A Faithful Benchmark for Information-Seeking Dialogue](https://mlsanity.com/paper/2204.10757/) (2022)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [A Dataset for Document Grounded Conversations](https://mlsanity.com/paper/1809.07358/) (2018)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data](https://mlsanity.com/paper/2010.05873/) (2020)
- [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://mlsanity.com/paper/2105.00071/) (2021)
- [Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features](https://mlsanity.com/paper/2107.06963/) (2021)
- [MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents](https://mlsanity.com/paper/2109.12595/) (2021)
- [A Controllable Model of Grounded Response Generation](https://mlsanity.com/paper/2005.00613/) (2020)
- [Logic2Text: High-Fidelity Natural Language Generation from Logical Forms](https://mlsanity.com/paper/2004.14579/) (2020)
- [Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation](https://mlsanity.com/paper/2110.05456/) (2021)
- [Focused Attention Improves Document-Grounded Generation](https://mlsanity.com/paper/2104.12714/) (2021)
- [Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization](https://mlsanity.com/paper/2109.06471/) (2021)
- [Neural Learning of One-of-Many Solutions for Combinatorial Problems in Structured Output Spaces](https://mlsanity.com/paper/2008.11990/) (2020)

## Built on by (most cited first, 11 of 11)

- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation](https://mlsanity.com/paper/2408.13586/) (2024)
- [Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG](https://mlsanity.com/paper/2502.08356/) (2025)
- [InfoBridge: Mutual Information estimation via Bridge Matching](https://mlsanity.com/paper/2502.01383/) (2025)
- [A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data](https://mlsanity.com/paper/2601.17717/) (2026)
- [Anti-LM Decoding for Zero-shot In-context Machine Translation](https://mlsanity.com/paper/2311.08324/) (2023)
- [Investigating Content Planning for Navigating Trade-offs in Knowledge-Grounded Dialogue](https://mlsanity.com/paper/2402.02077/) (2024)
- [Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding](https://mlsanity.com/paper/2606.22511/) (2026)
- [Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)](https://mlsanity.com/paper/2607.04223/) (2026)
- [DKL: Decoupled Knowledge Learning for Instruction-Tuned Language Models](https://mlsanity.com/paper/2609.02685/) (2026)
- [Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective](https://mlsanity.com/paper/2412.17787/) (2024)

## Abstract (arXiv)

A major concern in using deep learning based generative models for document-grounded dialogs is the potential generation of responses that are not textitfaithful to the underlying document. Existing automated metrics used for evaluating the faithfulness of response with respect to the grounding document measure the degree of similarity between the generated response and the document's content. However, these automated metrics are far from being well aligned with human judgments. Therefore, to improve the measurement of faithfulness, we propose a new metric that utilizes (Conditional) Point-wise Mutual Information (PMI) between the generated response and the source document, conditioned on the dialogue. PMI quantifies the extent to which the document influences the generated response -- with a higher PMI indicating a more faithful response. We build upon this idea to create a new decoding technique that incorporates PMI into the response generation process to predict more faithful responses. Our experiments on the BEGIN benchmark demonstrate an improved correlation of our metric with human evaluation. We also show that our decoding technique is effective in generating more faithful responses when compared to standard decoding techniques on a set of publicly available document-grounded dialog datasets.