mlsanityData to 7 Oct 2026

PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison

arXiv 2404.01015 ↗First posted 1 Apr 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses. To handle this, we propose PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. PairEval is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

43 of 55 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
31 more
  1. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  3. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  5. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  6. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  7. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  8. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  9. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  10. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  11. 2019Fairness in Recommendation Ranking through Pairwise Comparisonscited by 127
  12. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  13. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  14. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  15. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  16. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88
  17. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  18. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  19. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  20. 2020Improving Dialog Evaluation with a Multi-reference Adversarial Dataset and Large Scale Pretrainingcited by 59
  21. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  22. 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
  23. 2019Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddingscited by 52
  24. 2023A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluatorscited by 45
  25. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  26. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  27. 2021Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluationcited by 16
  28. 2020Evaluating Dialogue Generation Systems via Response Selectioncited by 9
  29. 2022Open-Domain Dialog Evaluation using Follow-Ups Likelihoodcited by 3
  30. 2022Pneg: Prompt-based Negative Response Generation for Dialogue Response Selection Taskcited by 3
  31. 2023DEnsity: Open-domain Dialogue Evaluation Metric using Density Estimationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 2.

11 papers.

  1. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  2. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  3. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  4. 2025LCES: Zero-shot Automated Essay Scoring via Pairwise Comparisons Using Large Language Modelscited by 8
  5. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  6. 2025Generalised Probabilistic Modelling and Improved Uncertainty Estimation in Comparative LLM-as-a-judgecited by 4
  7. 2025Exploring Persona Sentiment Sensitivity in Personalized Dialogue Generationcited by 3
  8. 2024Finetuning LLMs for Comparative Assessment Taskscited by 2
  9. 2025Evaluating LLM-Generated Versus Human-Authored Responses in Role-Play Dialoguescited by 1
  10. 2025LLM Optimization Unlocks Real-Time Pairwise Rerankingcited by 1
  11. 2026Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluationcited by 0