# Relevance-Promoting Language Model for Short-Text Conversation

arXiv: https://arxiv.org/abs/1911.11489 · HTML: https://mlsanity.com/paper/1911.11489/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Nov 2019.

## Built on by (most cited first, 3 of 3)

- [Generate, Prune, Select: A Pipeline for Counterspeech Generation against Online Hate Speech](https://mlsanity.com/paper/2106.01625/) (2021)
- [REAMsharp: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation](https://mlsanity.com/paper/2105.14488/) (2021)
- [Multi-party Response Generation with Relation Disentanglement](https://mlsanity.com/paper/2403.10827/) (2024)

## Abstract (arXiv)

Despite the effectiveness of sequence-to-sequence framework on the task of Short-Text Conversation (STC), the issue of under-exploitation of training data (i.e., the supervision signals from query text is textitignored) still remains unresolved. Also, the adopted textitmaximization-based decoding strategies, inclined to generating the generic responses or responses with repetition, are unsuited to the STC task. In this paper, we propose to formulate the STC task as a language modeling problem and tailor-make a training strategy to adapt a language model for response generation. To enhance generation performance, we design a relevance-promoting transformer language model, which performs additional supervised source attention after the self-attention to increase the importance of informative query tokens in calculating the token-level representation. The model further refines the query representation with relevance clues inferred from its multiple references during training. In testing, we adopt a textitrandomization-over-maximization strategy to reduce the generation of generic responses. Experimental results on a large Chinese STC dataset demonstrate the superiority of the proposed model on relevance metrics and diversity metrics.footnoteCode available at https://ai.tencent.com/ailab/nlp/dialogue/.