# Hierarchical Transformer for Task Oriented Dialog Systems

arXiv: https://arxiv.org/abs/2011.08067 · HTML: https://mlsanity.com/paper/2011.08067/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System](https://mlsanity.com/paper/2109.14739/) (2021)
- [Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understanding](https://mlsanity.com/paper/2105.12723/) (2021)
- [Grounding Dialogue Systems via Knowledge Graph Aware Decoding with Pre-trained Transformers](https://mlsanity.com/paper/2103.16289/) (2021)
- [MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generation](https://mlsanity.com/paper/2208.08629/) (2022)
- [Unsupervised Learning of Hierarchical Conversation Structure](https://mlsanity.com/paper/2205.12244/) (2022)
- [InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2310.08885/) (2023)
- [OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue](https://mlsanity.com/paper/2209.04595/) (2022)
- [DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learning](https://mlsanity.com/paper/2205.12662/) (2022)
- [Modeling Sequential Sentence Relation to Improve Cross-lingual Dense Retrieval](https://mlsanity.com/paper/2302.01626/) (2023)
- [A Deeper (Autoregressive) Approach to Non-Convergent Discourse Parsing](https://mlsanity.com/paper/2305.12510/) (2023)
- [CORAL: Contextual Response Retrievability Loss Function for Training Dialog Generation Models](https://mlsanity.com/paper/2205.10558/) (2022)
- [Frugal Prompting for Dialog Models](https://mlsanity.com/paper/2305.14919/) (2023)
- [Chat-Ghosting: A Comparative Study of Methods for Auto-Completion in Dialog Systems](https://mlsanity.com/paper/2507.05940/) (2025)
- [Local and Global Contexts for Conversation](https://mlsanity.com/paper/2401.17588/) (2024)

## Abstract (arXiv)

Generative models for dialog systems have gained much interest because of the recent success of RNN and Transformer based models in tasks like question answering and summarization. Although the task of dialog response generation is generally seen as a sequence-to-sequence (Seq2Seq) problem, researchers in the past have found it challenging to train dialog systems using the standard Seq2Seq models. Therefore, to help the model learn meaningful utterance and conversation level features, Sordoni et al. (2015b); Serban et al. (2016) proposed Hierarchical RNN architecture, which was later adopted by several other RNN based dialog systems. With the transformer-based models dominating the seq2seq problems lately, the natural question to ask is the applicability of the notion of hierarchy in transformer based dialog systems. In this paper, we propose a generalized framework for Hierarchical Transformer Encoders and show how a standard transformer can be morphed into any hierarchical encoder, including HRED and HIBERT like models, by using specially designed attention masks and positional encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models for task-oriented dialog systems through a wide range of experiments.