# CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systems

arXiv: https://arxiv.org/abs/2406.17300 · HTML: https://mlsanity.com/paper/2406.17300/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 63 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Revisiting Classifier Two-Sample Tests](https://mlsanity.com/paper/1610.06545/) (2016)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory](https://mlsanity.com/paper/1704.01074/) (2017)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems](https://mlsanity.com/paper/1701.03079/) (2017)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems](https://mlsanity.com/paper/2010.03994/) (2020)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [RankME: Reliable Human Ratings for Natural Language Generation](https://mlsanity.com/paper/1803.05928/) (2018)
- [DynaEval: Unifying Turn and Dialogue Level Evaluation](https://mlsanity.com/paper/2106.01112/) (2021)
- [Model-Powered Conditional Independence Test](https://mlsanity.com/paper/1709.06138/) (2017)
- [CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generation](https://mlsanity.com/paper/2105.08316/) (2021)
- [CCMI : Classifier based Conditional Mutual Information Estimation](https://mlsanity.com/paper/1906.01824/) (2019)

## Built on by (most cited first, 5 of 5)

- [On the Reliability of Large Language Models for Causal Discovery](https://mlsanity.com/paper/2407.19638/) (2024)
- [A Survey on Enhancing Causal Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2503.09326/) (2025)
- [Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation](https://mlsanity.com/paper/2507.06830/) (2025)
- [IRIS: An Iterative and Integrated Framework for Verifiable Causal Discovery in the Absence of Tabular Data](https://mlsanity.com/paper/2510.09217/) (2025)
- [Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue](https://mlsanity.com/paper/2608.26529/) (2026)

## Abstract (arXiv)

Automatically evaluating the quality of responses in open-domain dialogue systems is a challenging but crucial task. Current evaluation metrics often fail to align with human judgments, especially when assessing responses that are grammatically correct. To address this issue, we propose a novel metric, called CausalScore, which assesses the relevance of responses by measuring the causal strength between dialogue histories and responses. The causal strength is estimated by utilizing both unconditional dependence and conditional dependencies from the dialogue history to responses. We compare our metric with the existing competitive metrics in terms of their alignment with human judgements. Our experimental results demonstrate that CausalScore significantly surpasses existing state-of-the-art metrics by aligning better with human judgements. Additionally, we collect a new dialogue dataset CGDIALOG+ with human-annotated causal relations and a set of pairwise human judgements to facilitate the development of future automatic metrics.