# Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation

arXiv: https://arxiv.org/abs/2306.16650 · HTML: https://mlsanity.com/paper/2306.16650/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017)
- [Knowledge Bridging for Empathetic Dialogue Generation](https://mlsanity.com/paper/2009.09708/) (2020)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [Reasoning with Sarcasm by Reading In-between](https://mlsanity.com/paper/1805.02856/) (2018)
- [When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues](https://mlsanity.com/paper/2203.06419/) (2022)
- [R3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledge](https://mlsanity.com/paper/2004.13248/) (2020)
- [Multimodal Convolutional Neural Networks for Matching Image and Sentence](https://mlsanity.com/paper/1504.06063/) (2015)
- [Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation](https://mlsanity.com/paper/2112.04873/) (2021)
- [KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation](https://mlsanity.com/paper/2101.00419/) (2021)
- [Sarcasm SIGN: Interpreting Sarcasm with Sentiment Based Monolingual Machine Translation](https://mlsanity.com/paper/1704.06836/) (2017)
- [The Role of Conversation Context for Sarcasm Detection in Online Interactions](https://mlsanity.com/paper/1707.06226/) (2017)
- [KinGDOM: Knowledge-Guided DOMain adaptation for sentiment analysis](https://mlsanity.com/paper/2005.00791/) (2020)
- [Deep Multi-Task Model for Sarcasm Detection and Sentiment Analysis in Arabic Language](https://mlsanity.com/paper/2106.12488/) (2021)

## Built on by (most cited first, 19 of 19)

- [Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practices](https://mlsanity.com/paper/2408.09169/) (2024)
- [Can Large Multimodal Models Uncover Deep Semantics Behind Images?](https://mlsanity.com/paper/2402.11281/) (2024)
- [Sentiment-enhanced Graph-based Sarcasm Explanation in Dialogue](https://mlsanity.com/paper/2402.03658/) (2024)
- [On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs](https://mlsanity.com/paper/2410.12600/) (2024)
- [PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension](https://mlsanity.com/paper/2412.11906/) (2024)
- [Can Large Vision-Language Models Understand Multimodal Sarcasm?](https://mlsanity.com/paper/2508.03654/) (2025)
- [InterCLIP-MEP: Interactive CLIP and Memory-Enhanced Predictor for Multi-modal Sarcasm Detection](https://mlsanity.com/paper/2406.16464/) (2024)
- [Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models](https://mlsanity.com/paper/2503.12149/) (2025)
- [A Unified Hallucination Mitigation Framework for Large Vision-Language Models](https://mlsanity.com/paper/2409.16494/) (2024)
- [Target-Augmented Shared Fusion-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2502.07391/) (2025)
- [VK-G2T: Vision and Context Knowledge enhanced Gloss2Text](https://mlsanity.com/paper/2312.10210/) (2023)
- [Alert-ME: An Explainability-Driven Defense Against Adversarial Examples in Transformer-Based Text Classification](https://mlsanity.com/paper/2307.01225/) (2023)
- [Augmented Vision-Language Models: A Systematic Review](https://mlsanity.com/paper/2507.22933/) (2025)
- [Graph Integrated Multimodal Concept Bottleneck Model](https://mlsanity.com/paper/2510.00701/) (2025)
- [Nek Minit: Harnessing Pragmatic Metacognitive Prompting for Explainable Sarcasm Detection of Australian and Indian English](https://mlsanity.com/paper/2505.15095/) (2025)
- [ViSP: A PPO-Driven Framework for Sarcasm Generation with Contrastive Learning](https://mlsanity.com/paper/2507.09482/) (2025)
- [MuVaC: A Variational Causal Framework for Multimodal Sarcasm Understanding in Dialogues](https://mlsanity.com/paper/2601.20451/) (2026)
- [Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding](https://mlsanity.com/paper/2606.03604/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which aims to generate a natural language sentence for a multimodal social post (an image as well as its caption) to explain why it contains sarcasm. Although the existing pioneer study has achieved great success with the BART backbone, it overlooks the gap between the visual feature space and the decoder semantic space, the object-level metadata of the image, as well as the potential external knowledge. To solve these limitations, in this work, we propose a novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme, named TEAM. In particular, TEAM extracts the object-level semantic meta-data instead of the traditional global visual features from the input image. Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge concepts for the input text and the extracted object meta-data. Thereafter, TEAM introduces a multi-source semantic graph that comprehensively characterize the multi-source (i.e., caption, object meta-data, external knowledge) semantic relations to facilitate the sarcasm reasoning. Extensive experiments on a public released dataset MORE verify the superiority of our model over cutting-edge methods.