# Position Debiasing Fine-Tuning for Causal Perception in Long-Term Dialogue

arXiv: https://arxiv.org/abs/2406.02002 · HTML: https://mlsanity.com/paper/2406.02002/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Invariant Rationalization](https://mlsanity.com/paper/2003.09772/) (2020)
- [Long Time No See! Open-Domain Conversation with Long-Term Persona Memory](https://mlsanity.com/paper/2203.05797/) (2022)
- [Attention Sorting Combats Recency Bias In Long Context Language Models](https://mlsanity.com/paper/2310.01427/) (2023)
- [Look at the First Sentence: Position Bias in Question Answering](https://mlsanity.com/paper/2004.14602/) (2020)
- [Split and Merge: Aligning Position Biases in LLM-based Evaluators](https://mlsanity.com/paper/2310.01432/) (2023)
- [Efficient Long-Text Understanding with Short-Text Models](https://mlsanity.com/paper/2208.00748/) (2022)
- [Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models](https://mlsanity.com/paper/2308.15022/) (2023)
- [Prompted LLMs as Chatbot Modules for Long Open-domain Conversation](https://mlsanity.com/paper/2305.04533/) (2023)
- [CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior](https://mlsanity.com/paper/2205.14140/) (2022)
- [Improving Personality Consistency in Conversation by Persona Extending](https://mlsanity.com/paper/2208.10816/) (2022)
- [MIRACLE: Towards Personalized Dialogue Generation with Latent-Space Multiple Personal Attribute Control](https://mlsanity.com/paper/2310.18342/) (2023)
- [Target Guided Emotion Aware Chat Machine](https://mlsanity.com/paper/2011.07432/) (2020)
- [Less is More: Mitigate Spurious Correlations for Open-Domain Dialogue Response Generation Models by Causal Discovery](https://mlsanity.com/paper/2303.01962/) (2023)
- [Instruction Position Matters in Sequence Generation with Large Language Models](https://mlsanity.com/paper/2308.12097/) (2023)
- [CFL: Causally Fair Language Models Through Token-level Attribute Controlled Generation](https://mlsanity.com/paper/2306.00374/) (2023)
- [Incorporating Causal Analysis into Diversified and Logical Response Generation](https://mlsanity.com/paper/2209.09482/) (2022)
- [Technical Report: Impact of Position Bias on Language Models in Token Classification](https://mlsanity.com/paper/2304.13567/) (2023)

## Abstract (arXiv)

The core of the dialogue system is to generate relevant, informative, and human-like responses based on extensive dialogue history. Recently, dialogue generation domain has seen mainstream adoption of large language models (LLMs), due to its powerful capability in generating utterances. However, there is a natural deficiency for such models, that is, inherent position bias, which may lead them to pay more attention to the nearby utterances instead of causally relevant ones, resulting in generating irrelevant and generic responses in long-term dialogue. To alleviate such problem, in this paper, we propose a novel method, named Causal Perception long-term Dialogue framework (CPD), which employs perturbation-based causal variable discovery method to extract casually relevant utterances from the dialogue history and enhances model causal perception during fine-tuning. Specifically, a local-position awareness method is proposed in CPD for inter-sentence position correlation elimination, which helps models extract causally relevant utterances based on perturbations. Then, a casual-perception fine-tuning strategy is also proposed, to enhance the capability of discovering the causal invariant factors, by differently perturbing causally relevant and non-casually relevant ones for response generation. Experimental results on two datasets prove that our proposed method can effectively alleviate the position bias for multiple LLMs and achieve significant progress compared with existing baselines.