mlsanityData to 7 Oct 2026

Rethinking and Refining the Distinct Metric

arXiv 2202.13587 ↗First posted 28 Feb 2022cs.CL cs.AIcited by 27

Not on the map. Why?

Abstract (arXiv)

Distinct-n scoreciteLi2016 is a widely used automatic metric for evaluating diversity in language generation tasks. However, we observed that the original approach for calculating distinct scores has evident biases that tend to assign higher penalties to longer sequences. We refine the calculation of distinct scores by scaling the number of distinct tokens based on their expectations. We provide both empirical and theoretical evidence to show that our method effectively removes the biases existing in the original distinct score. Our experiments show that our proposed metric, textitExpectation-Adjusted Distinct (EAD), correlates better with human judgment in evaluating response diversity. To foster future research, we provide an example implementation at urlhttps://github.com/lsy641/Expectation-Adjusted-Distinct.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 37 of its 60 references that are in the map. First posted on arXiv 28 Feb 2022.

Builds on

37 of 60 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2018Hierarchical Neural Story Generationcited by 1,182
  7. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  8. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  9. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  10. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  11. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  12. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
25 more
  1. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  2. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  3. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  4. 2021Towards Emotional Support Dialog Systemscited by 294
  5. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  6. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  7. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  8. 2017RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systemscited by 88
  9. 2020A Large-Scale Chinese Short-Text Conversation Datasetcited by 77
  10. 2019A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Datacited by 74
  11. 2019Personalized Dialogue Generation with Diversified Traitscited by 73
  12. 2021OpenMEVA: A Benchmark for Evaluating Open-ended Story Generation Metricscited by 61
  13. 2020Vocabulary Learning via Optimal Transport for Neural Machine Translationcited by 58
  14. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34
  15. 2021EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Trainingcited by 34
  16. 2015A Survey of Available Corpora for Building Data-Driven Dialogue Systemscited by 33
  17. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  18. 2019Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Losscited by 29
  19. 2021Diversifying Dialog Generation via Adaptive Label Smoothingcited by 25
  20. 2019Negative Training for Neural Dialogue Response Generationcited by 24
  21. 2020Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Datacited by 20
  22. 2020Stylized Dialogue Response Generation Using Stylized Unpaired Textscited by 16
  23. 2021Transferable Persona-Grounded Dialogues via Grounded Minimal Editscited by 10
  24. 2020F^2-Softmax: Diversifying Neural Text Generation via Frequency Factorized Softmaxcited by 8
  25. 2021Semantic-Enhanced Explainable Finetuning for Open-Domain Dialoguescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 2; 2025: 11; 2026: 11.

27 papers.

Sort:

By area:

  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2022PAL: Persona-Augmented Emotional Support Conversation Generationcited by 39
  3. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  4. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  5. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  6. 2022Factual and Informative Review Generation for Explainable Recommendationcited by 14
  7. 2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 14
  8. 2025G2: Guided Generation for Enhanced Output Diversity in LLMscited by 14
  9. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 10
  10. 2026Where does output diversity collapse in post-training?cited by 10
  11. 2025Optimizing Diversity and Quality through Base-Aligned Model Collaborationcited by 8
  12. 2025MAGneT: Coordinated Multi-Agent Generation of Synthetic Multi-Turn Mental Health Counseling Sessionscited by 6
15 more
  1. 2026DPWriter: Reinforcement Learning with Diverse Planning Branching for Creative Writingcited by 6
  2. 2025Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimizationcited by 3
  3. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  4. 2024Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Modelscited by 2
  5. 2025DiffSampling: Enhancing Diversity and Accuracy in Neural Text Generationcited by 2
  6. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  7. 2025Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generationcited by 1
  8. 2026D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decodingcited by 1
  9. 2026The Limits of Automatic Evaluation of Creativity in Large Language Modelscited by 1
  10. 2025Following the TRACE: A Structured Path to Empathetic Response Generation with Multi-Agent Modelscited by 0
  11. 2026On Temperature-Constrained Non-Deterministic Machine Translation: Potential and Evaluationcited by 0
  12. 2026Detecting RLVR Training Data via Structural Convergence of Reasoningcited by 0
  13. 2026CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversitycited by 0
  14. 2026FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Modelscited by 0
  15. 2026A Harness for Synthesizing Diverse Naturalistic Full-Duplex Conversationscited by 0