# Similarity Based Label Smoothing For Dialogue Generation

arXiv: https://arxiv.org/abs/2107.11481 · HTML: https://mlsanity.com/paper/2107.11481/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 23 Jul 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Improved Natural Language Generation via Loss Truncation](https://mlsanity.com/paper/2004.14589/) (2020)
- [Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight](https://mlsanity.com/paper/2004.02594/) (2020)
- [Improving Neural Response Diversity with Frequency-Aware Cross-Entropy Loss](https://mlsanity.com/paper/1902.09191/) (2019)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [Negative Training for Neural Dialogue Response Generation](https://mlsanity.com/paper/1903.02134/) (2019)
- [Improving Text Generation with Student-Forcing Optimal Transport](https://mlsanity.com/paper/2010.05994/) (2020)

## Built on by (most cited first, 2 of 2)

- [Rethinking Label Smoothing on Multi-hop Question Answering](https://mlsanity.com/paper/2212.09512/) (2022)
- [Reprint: a randomized extrapolation based on principal components for data augmentation](https://mlsanity.com/paper/2204.12024/) (2022)

## Abstract (arXiv)

Generative neural conversational systems are generally trained with the objective of minimizing the entropy loss between the training "hard" targets and the predicted logits. Often, performance gains and improved generalization can be achieved by using regularization techniques like label smoothing, which converts the training "hard" targets to "soft" targets. However, label smoothing enforces a data independent uniform distribution on the incorrect training targets, which leads to an incorrect assumption of equi-probable incorrect targets for each correct target. In this paper we propose and experiment with incorporating data dependent word similarity based weighing methods to transforms the uniform distribution of the incorrect target probabilities in label smoothing, to a more natural distribution based on semantics. We introduce hyperparameters to control the incorrect target distribution, and report significant performance gains over networks trained using standard label smoothing based loss, on two standard open domain dialogue corpora.