# A non-hierarchical attention network with modality dropout for textual response generation in multimodal dialogue systems

arXiv: https://arxiv.org/abs/2110.09702 · HTML: https://mlsanity.com/paper/2110.09702/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 14 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 4 of 4)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)

## Built on by (most cited first, 7 of 7)

- [Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2304.10756/) (2023)
- [ModDrop++: A Dynamic Filter Network with Intra-subject Co-training for Multiple Sclerosis Lesion Segmentation with Missing Modalities](https://mlsanity.com/paper/2203.04959/) (2022)
- [Automatic Context Pattern Generation for Entity Set Expansion](https://mlsanity.com/paper/2207.08087/) (2022)
- [Towards Attribute-Entangled Controllable Text Generation: A Pilot Study of Blessing Generation](https://mlsanity.com/paper/2210.16557/) (2022)
- [From Retrieval to Generation: Efficient and Effective Entity Set Expansion](https://mlsanity.com/paper/2304.03531/) (2023)
- [Scene-Aware Prompt for Multi-modal Dialogue Understanding and Generation](https://mlsanity.com/paper/2207.01823/) (2022)
- [Application of frozen large-scale models to multimodal task-oriented dialogue](https://mlsanity.com/paper/2310.00845/) (2023)

## Abstract (arXiv)

Existing text- and image-based multimodal dialogue systems use the traditional Hierarchical Recurrent Encoder-Decoder (HRED) framework, which has an utterance-level encoder to model utterance representation and a context-level encoder to model context representation. Although pioneer efforts have shown promising performances, they still suffer from the following challenges: (1) the interaction between textual features and visual features is not fine-grained enough. (2) the context representation can not provide a complete representation for the context. To address the issues mentioned above, we propose a non-hierarchical attention network with modality dropout, which abandons the HRED framework and utilizes attention modules to encode each utterance and model the context representation. To evaluate our proposed model, we conduct comprehensive experiments on a public multimodal dialogue dataset. Automatic and human evaluation demonstrate that our proposed model outperforms the existing methods and achieves state-of-the-art performance.