# C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation

arXiv: https://arxiv.org/abs/2306.15245 · HTML: https://mlsanity.com/paper/2306.15245/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 27 Jun 2023.

## Builds on (most cited first, 15 of 15)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation](https://mlsanity.com/paper/2005.00456/) (2020)
- [Unsupervised Evaluation of Interactive Dialog with DialoGPT](https://mlsanity.com/paper/2006.12719/) (2020)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Dialogue Response Ranking Training with Large-Scale Human Feedback Data](https://mlsanity.com/paper/2009.06978/) (2020)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [Deconstruct to Reconstruct a Configurable Evaluation Metric for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2011.00483/) (2020)
- [Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems](https://mlsanity.com/paper/1911.01456/) (2019)
- [Enhanced Knowledge Selection for Grounded Dialogues via Document Semantic Graphs](https://mlsanity.com/paper/2206.07296/) (2022)
- [Interaction Information for Causal Inference: The Case of Directed Triangle](https://mlsanity.com/paper/1701.08868/) (2017)

## Built on by (most cited first, 5 of 5)

- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Measuring the Robustness of Reference-Free Dialogue Evaluation Systems](https://mlsanity.com/paper/2501.06728/) (2025)
- [PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement](https://mlsanity.com/paper/2603.13796/) (2026)
- [Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs](https://mlsanity.com/paper/2608.26641/) (2026)
- [Not Every Correction Helps: Gain-Guided Continual Test-Time Adaptation](https://mlsanity.com/paper/2609.36655/) (2026)

## Abstract (arXiv)

Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system. Consequently, they often correlate poorly with human evaluations. To address this issue, we propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve a relative 62.6% higher Spearman correlation on average for the FED evaluation metric. Our code is publicly available at https://github.com/renll/C-PMI.