# Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmark

arXiv: https://arxiv.org/abs/2109.01839 · HTML: https://mlsanity.com/paper/2109.01839/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- [SemEval-2020 Task 8: Memotion Analysis -- The Visuo-Lingual Metaphor!](https://mlsanity.com/paper/2008.03781/) (2020)
- [A Large-Scale Chinese Short-Text Conversation Dataset](https://mlsanity.com/paper/2008.03946/) (2020)
- [Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation](https://mlsanity.com/paper/1701.08251/) (2017)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [The PhotoBook Dataset: Building Common Ground through Visually-Grounded Dialogue](https://mlsanity.com/paper/1906.01530/) (2019)
- [Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog](https://mlsanity.com/paper/2002.00163/) (2020)
- [Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog](https://mlsanity.com/paper/2003.04679/) (2020)
- [Multimodal Emoji Prediction](https://mlsanity.com/paper/1803.02392/) (2018)
- [Are Emojis Predictable?](https://mlsanity.com/paper/1702.07285/) (2017)
- [Understanding Chat Messages for Sticker Recommendation in Messaging Apps](https://mlsanity.com/paper/1902.02704/) (2019)
- [Neural Emoji Recommendation in Dialogue Systems](https://mlsanity.com/paper/1612.04609/) (2016)
- [Semantic Search of Memes on Twitter](https://mlsanity.com/paper/2002.01462/) (2020)

## Built on by (most cited first, 12 of 12)

- [OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation](https://mlsanity.com/paper/2411.18499/) (2024)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)
- [C3KG: A Chinese Commonsense Conversation Knowledge Graph](https://mlsanity.com/paper/2204.02549/) (2022)
- [Sticker820K: Empowering Interactive Retrieval with Stickers](https://mlsanity.com/paper/2306.06870/) (2023)
- [Towards Building an Open-Domain Dialogue System Incorporated with Internet Memes](https://mlsanity.com/paper/2203.03835/) (2022)
- [Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selection](https://mlsanity.com/paper/2511.17587/) (2025)
- [Selecting Stickers in Open-Domain Dialogue through Multitask Learning](https://mlsanity.com/paper/2209.07697/) (2022)
- [RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame Animated Sticker Generation](https://mlsanity.com/paper/2503.17735/) (2025)
- [GSE: Evaluating Sticker Visual Semantic Similarity via a General Sticker Encoder](https://mlsanity.com/paper/2511.04977/) (2025)
- [Memes-as-Replies: Can Models Select Humorous Manga Panel Responses?](https://mlsanity.com/paper/2602.15842/) (2026)
- [SEAL: Semantic-aware Single-image Sticker Personalization with a Large-scale Sticker-tag Dataset](https://mlsanity.com/paper/2604.26883/) (2026)

## Abstract (arXiv)

As a kind of new expression elements, Internet memes are popular and extensively used in online chatting scenarios since they manage to make dialogues vivid, moving, and interesting. However, most current dialogue researches focus on text-only dialogue tasks. In this paper, we propose a new task named as Meme incorporated Open-domain Dialogue (MOD). Compared to previous dialogue tasks, MOD is much more challenging since it requires the model to understand the multimodal elements as well as the emotions behind them. To facilitate the MOD research, we construct a large-scale open-domain multimodal dialogue dataset incorporating abundant Internet memes into utterances. The dataset consists of ~45K Chinese conversations with ~606K utterances. Each conversation contains about 13 utterances with about 4 Internet memes on average and each utterance equipped with an Internet meme is annotated with the corresponding emotion. In addition, we present a simple and effective method, which utilizes a unified generation network to solve the MOD task. Experimental results demonstrate that our method trained on the proposed corpus is able to achieve expressive communication including texts and memes. The corpus and models have been publicly available at https://github.com/lizekang/DSTC10-MOD.