# Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue

arXiv: https://arxiv.org/abs/2302.05888 · HTML: https://mlsanity.com/paper/2302.05888/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Internet-Augmented Dialogue Generation](https://mlsanity.com/paper/2107.07566/) (2021)
- [TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents](https://mlsanity.com/paper/1901.08149/) (2019)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Dialogue Natural Language Inference](https://mlsanity.com/paper/1811.00671/) (2018)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models](https://mlsanity.com/paper/2010.00840/) (2020)
- [Dialogue Response Ranking Training with Large-Scale Human Feedback Data](https://mlsanity.com/paper/2009.06978/) (2020)
- [What Context Features Can Transformer Language Models Use?](https://mlsanity.com/paper/2106.08367/) (2021)
- [Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study](https://mlsanity.com/paper/1906.01603/) (2019)
- [Generate, Delete and Rewrite: A Three-Stage Framework for Improving Persona Consistency of Dialogue Generation](https://mlsanity.com/paper/2004.07672/) (2020)
- [A Model-Agnostic Data Manipulation Method for Persona-based Dialogue Generation](https://mlsanity.com/paper/2204.09867/) (2022)
- [A Three-Stage Learning Framework for Low-Resource Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2109.04096/) (2021)
- [Transferable Persona-Grounded Dialogues via Grounded Minimal Edits](https://mlsanity.com/paper/2109.07713/) (2021)
- [Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue](https://mlsanity.com/paper/2110.06905/) (2021)

## Built on by (most cited first, 1 of 1)

- [Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting](https://mlsanity.com/paper/2305.14755/) (2023)

## Abstract (arXiv)

With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems. The traditional techniques treat knowledge as part of the input sequence for the dialogue system, prepending a set of knowledge statements in front of dialogue history. However, such a mechanism forces knowledge sets to be concatenated in an ordered manner, making models implicitly pay imbalanced attention to the sets during training. In this paper, we first investigate how the order of the knowledge set can influence autoregressive dialogue systems' responses. We conduct experiments on two commonly used dialogue datasets with two types of transformer-based models and find that models view the input knowledge unequally. To this end, we propose a simple and novel technique to alleviate the order effect by modifying the position embeddings of knowledge input in these models. With the proposed position embedding method, the experimental results show that each knowledge statement is uniformly considered to generate responses.