# History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System

arXiv: https://arxiv.org/abs/2302.00907 · HTML: https://mlsanity.com/paper/2302.00907/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 2 Feb 2023.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models](https://mlsanity.com/paper/1507.04808/) (2015)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Challenges in Building Intelligent Open-domain Dialog Systems](https://mlsanity.com/paper/1905.05709/) (2019)
- [From Eliza to XiaoIce: Challenges and Opportunities with Social Chatbots](https://mlsanity.com/paper/1801.01957/) (2018)
- [Towards Persona-Based Empathetic Conversational Models](https://mlsanity.com/paper/2004.12316/) (2020)
- [Pointing the Unknown Words](https://mlsanity.com/paper/1603.08148/) (2016)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions](https://mlsanity.com/paper/2006.12442/) (2020)
- [Learning Implicit User Profiles for Personalized Retrieval-Based Chatbot](https://mlsanity.com/paper/2108.07935/) (2021)
- [Content Selection Network for Document-grounded Retrieval-based Chatbots](https://mlsanity.com/paper/2101.08426/) (2021)
- [A Spoken Dialogue System for Spatial Question Answering in a Physical Blocks World](https://mlsanity.com/paper/1911.02524/) (2019)

## Built on by (most cited first, 15 of 15)

- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models](https://mlsanity.com/paper/2308.15022/) (2023)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversations](https://mlsanity.com/paper/2402.11975/) (2024)
- [Gradient based Feature Attribution in Explainable AI: A Technical Review](https://mlsanity.com/paper/2403.10415/) (2024)
- [A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models](https://mlsanity.com/paper/2504.07070/) (2025)
- [NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splatting](https://mlsanity.com/paper/2507.12621/) (2025)
- [Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation](https://mlsanity.com/paper/2310.15415/) (2023)
- [Context-Aware LLM Translation System Using Conversation Summarization and Dialogue History](https://mlsanity.com/paper/2410.16775/) (2024)
- [May I Ask a Follow-up Question? Understanding the Benefits of Conversations in Neural Network Explainability](https://mlsanity.com/paper/2309.13965/) (2023)
- [Enhancing Long-Term Memory using Hierarchical Aggregate Tree for Retrieval Augmented Generation](https://mlsanity.com/paper/2406.06124/) (2024)
- [A Framework for Longitudinal Health AI Agents](https://mlsanity.com/paper/2604.12019/) (2026)
- [FAQ-Gen: An automated system to generate domain-specific FAQs to aid content comprehension](https://mlsanity.com/paper/2402.05812/) (2024)
- [MT-OSC: Path for LLMs that Get Lost in Multi-Turn Conversation](https://mlsanity.com/paper/2604.08782/) (2026)
- [Triple-Encoders: Representations That Fire Together, Wire Together](https://mlsanity.com/paper/2402.12332/) (2024)

## Abstract (arXiv)

With the evolution of pre-trained language models, current open-domain dialogue systems have achieved great progress in conducting one-session conversations. In contrast, Multi-Session Conversation (MSC), which consists of multiple sessions over a long term with the same user, is under-investigated. In this paper, we propose History-Aware Hierarchical Transformer (HAHT) for multi-session open-domain dialogue. HAHT maintains a long-term memory of history conversations and utilizes history information to understand current conversation context and generate well-informed and context-relevant responses. Specifically, HAHT first encodes history conversation sessions hierarchically into a history memory. Then, HAHT leverages historical information to facilitate the understanding of the current conversation context by encoding the history memory together with the current context with attention-based mechanisms. Finally, to explicitly utilize historical information, HAHT uses a history-aware response generator that switches between a generic vocabulary and a history-aware vocabulary. Experimental results on a large-scale MSC dataset suggest that the proposed HAHT model consistently outperforms baseline models. Human evaluation results support that HAHT generates more human-like, context-relevant and history-relevant responses than baseline models.