mlsanityData to 7 Oct 2026

Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation

arXiv 2104.05224 ↗First posted 12 Apr 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Human ratings are one of the most prevalent methods to evaluate the performance of natural language processing algorithms. Similarly, it is common to measure the quality of sentences generated by a natural language generation model using human raters. In this paper, we argue for exploring the use of subjective evaluations within the process of training language generation models in a multi-task learning setting. As a case study, we use a crowd-authored dialogue corpus to fine-tune six different language generation models. Two of these models incorporate multi-task learning and use subjective ratings of lines as part of an explicit learning goal. A human evaluation of the generated dialogue lines reveals that utterances generated by the multi-tasking models were subjectively rated as the most typical, most moving the conversation forward, and least offensive. Based on these promising first results, we discuss future research directions for incorporating subjective human evaluations into language model training and to hence keep the human user in the loop during the development process.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 59 references that are in the map. First posted on arXiv 12 Apr 2021.

Builds on

30 of 59 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  8. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  10. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  11. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  12. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
18 more
  1. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  2. 2019TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agentscited by 188
  3. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  4. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  5. 2017ParlAI: A Dialog Research Software Platformcited by 139
  6. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  7. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  8. 2017Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasmcited by 117
  9. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  10. 2019MoEL: Mixture of Empathetic Listenerscited by 91
  11. 2020You Impress Me: Dialogue Generation via Mutual Persona Perceptioncited by 85
  12. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  13. 2017MojiTalk: Generating Emotional Responses at Scalecited by 64
  14. 2020CDL: Curriculum Dual Learning for Emotion-Controllable Response Generationcited by 43
  15. 2020EmpTransfo: A Multi-head Transformer Architecture for Creating Empathetic Dialog Systemscited by 19
  16. 2016A Context-aware Natural Language Generator for Dialogue Systemscited by 13
  17. 2020An Empirical Investigation of Pre-Trained Transformer Language Models for Open-Domain Dialogue Generationcited by 7
  18. 2019"Can you say more about the location?" The Development of a Pedagogical Reference Resolution Agentcited by 1

Built on by

Not cited yet.