# deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets

arXiv: https://arxiv.org/abs/1506.06863 · HTML: https://mlsanity.com/paper/1506.06863/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2015.

## Built on by (most cited first, 21 of 21)

- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [A Comprehensive Assessment of Dialog Evaluation Metrics](https://mlsanity.com/paper/2106.03706/) (2021)
- [Shades of BLEU, Flavours of Success: The Case of MultiWOZ](https://mlsanity.com/paper/2106.05555/) (2021)
- [InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation](https://mlsanity.com/paper/2112.01589/) (2021)
- [Why is constrained neural language generation particularly challenging?](https://mlsanity.com/paper/2206.05395/) (2022)
- [Towards Automatic Evaluation of Dialog Systems: A Model-Free Off-Policy Evaluation Approach](https://mlsanity.com/paper/2102.10242/) (2021)
- [MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering](https://mlsanity.com/paper/2509.12405/) (2025)
- [Language Model Augmented Relevance Score](https://mlsanity.com/paper/2108.08485/) (2021)
- [Meta-evaluation of Conversational Search Evaluation Metrics](https://mlsanity.com/paper/2104.13453/) (2021)
- [Improving Automated Evaluation of Open Domain Dialog via Diverse Reference Augmentation](https://mlsanity.com/paper/2106.02833/) (2021)
- [Evaluation of Question Answering Systems: Complexity of judging a natural language](https://mlsanity.com/paper/2209.12617/) (2022)
- [Learning to Improve Code Efficiency](https://mlsanity.com/paper/2208.05297/) (2022)
- [Multimodal Retrieval-Augmented Generation with Large Language Models for Medical VQA](https://mlsanity.com/paper/2510.13856/) (2025)
- [How to Evaluate Your Dialogue Models: A Review of Approaches](https://mlsanity.com/paper/2108.01369/) (2021)
- [Calibrating Translation Decoding with Quality Estimation on LLMs](https://mlsanity.com/paper/2504.19044/) (2025)
- [On the Use of Linguistic Features for the Evaluation of Generative Dialogue Systems](https://mlsanity.com/paper/2104.06335/) (2021)
- [How to Choose How to Choose Your Chatbot: A Massively Multi-System MultiReference Data Set for Dialog Metric Evaluation](https://mlsanity.com/paper/2305.14533/) (2023)
- [Should we tweet this? Generative response modeling for predicting reception of public health messaging on Twitter](https://mlsanity.com/paper/2204.04353/) (2022)
- [Improving a sequence-to-sequence nlp model using a reinforcement learning policy algorithm](https://mlsanity.com/paper/2212.14117/) (2022)
- [Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systems](https://mlsanity.com/paper/2401.02256/) (2024)
- [WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models](https://mlsanity.com/paper/2404.14567/) (2024)

## Abstract (arXiv)

We introduce Discriminative BLEU (deltaBLEU), a novel metric for intrinsic evaluation of generated text in tasks that admit a diverse range of possible outputs. Reference strings are scored for quality by human raters on a scale of [-1, +1] to weight multi-reference BLEU. In tasks involving generation of conversational responses, deltaBLEU correlates reasonably with human judgments and outperforms sentence-level and IBM BLEU in terms of both Spearman's rho and Kendall's tau.