# XMeCap: Meme Caption Generation with Sub-Image Adaptability

arXiv: https://arxiv.org/abs/2407.17152 · HTML: https://mlsanity.com/paper/2407.17152/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 24 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks](https://mlsanity.com/paper/2206.08916/) (2022)
- [Scaling Language-Image Pre-training via Masking](https://mlsanity.com/paper/2212.00794/) (2022)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [A Study of the Attention Abnormality in Trojaned BERTs](https://mlsanity.com/paper/2205.08305/) (2022)
- [Describing like humans: on diversity in image captioning](https://mlsanity.com/paper/1903.12020/) (2019)
- [Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models](https://mlsanity.com/paper/2407.04121/) (2024)
- [Contextual Hourglass Network for Semantic Segmentation of High Resolution Aerial Imagery](https://mlsanity.com/paper/1810.12813/) (2018)
- [Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformers](https://mlsanity.com/paper/2406.12199/) (2024)
- [Attention-Enhancing Backdoor Attacks Against BERT-based Models](https://mlsanity.com/paper/2310.14480/) (2023)
- [Confidence Trigger Detection: Accelerating Real-time Tracking-by-detection Systems](https://mlsanity.com/paper/1902.00615/) (2019)
- [A Comparative Study on Enhancing Prediction in Social Network Advertisement through Data Augmentation](https://mlsanity.com/paper/2404.13812/) (2024)
- [MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization](https://mlsanity.com/paper/2407.04118/) (2024)
- [Enhance Image-to-Image Generation with LLaVA-generated Prompts](https://mlsanity.com/paper/2406.01956/) (2024)
- [Grow-and-Clip: Informative-yet-Concise Evidence Distillation for Answer Explanation](https://mlsanity.com/paper/2201.05088/) (2022)
- [Caption Enriched Samples for Improving Hateful Memes Detection](https://mlsanity.com/paper/2109.10649/) (2021)
- [Can Pre-trained Language Models Understand Chinese Humor?](https://mlsanity.com/paper/2407.04105/) (2024)
- [Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models](https://mlsanity.com/paper/2407.11033/) (2024)
- [Beyond Single-Event Extraction: Towards Efficient Document-Level Multi-Event Argument Extraction](https://mlsanity.com/paper/2405.01884/) (2024)
- [Dank Learning: Generating Memes Using Deep Neural Networks](https://mlsanity.com/paper/1806.04510/) (2018)
- [Enhancing Document-level Event Argument Extraction with Contextual Clues and Role Relevance](https://mlsanity.com/paper/2310.05991/) (2023)
- [memeBot: Towards Automatic Image Meme Generation](https://mlsanity.com/paper/2004.14571/) (2020)
- [M2H2: A Multimodal Multiparty Hindi Dataset For Humor Recognition in Conversations](https://mlsanity.com/paper/2108.01260/) (2021)
- [Memeify: A Large-Scale Meme Generation System](https://mlsanity.com/paper/1910.12279/) (2019)
- [Multi-modal application: Image Memes Generation](https://mlsanity.com/paper/2112.01651/) (2021)

## Built on by (most cited first, 14 of 14)

- [NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training](https://mlsanity.com/paper/2409.09582/) (2024)
- [Recent Advancement of Emotion Cognition in Large Language Models](https://mlsanity.com/paper/2409.13354/) (2024)
- [HOTVCOM: Generating Buzzworthy Comments for Videos](https://mlsanity.com/paper/2409.15196/) (2024)
- [Journey into Automation: Image-Derived Pavement Texture Extraction and Evaluation](https://mlsanity.com/paper/2501.02414/) (2025)
- [What Media Frames Reveal About Stance: A Dataset and Study about Memes in Climate Change Discourse](https://mlsanity.com/paper/2505.16592/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [Enhancing Exchange Rate Forecasting with Explainable Deep Learning Models](https://mlsanity.com/paper/2410.19241/) (2024)
- [From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme](https://mlsanity.com/paper/2512.24555/) (2025)
- [MER-Bench: A Comprehensive Benchmark for Multimodal Meme Reappraisal](https://mlsanity.com/paper/2603.15020/) (2026)
- [Pre-trained Graphformer-based Ranking at Web-scale Search (Extended Abstract)](https://mlsanity.com/paper/2409.16590/) (2024)
- [Generative Pre-trained Ranking Model with Over-parameterization at Web-Scale (Extended Abstract)](https://mlsanity.com/paper/2409.16594/) (2024)
- [Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts](https://mlsanity.com/paper/2604.18091/) (2026)
- [Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges](https://mlsanity.com/paper/2607.19011/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

Humor, deeply rooted in societal meanings and cultural details, poses a unique challenge for machines. While advances have been made in natural language processing, real-world humor often thrives in a multi-modal context, encapsulated distinctively by memes. This paper poses a particular emphasis on the impact of multi-images on meme captioning. After that, we introduce the textscXMeCap framework, a novel approach that adopts supervised fine-tuning and reinforcement learning based on an innovative reward model, which factors in both global and local similarities between visuals and text. Our results, benchmarked against contemporary models, manifest a marked improvement in caption generation for both single-image and multi-image memes, as well as different meme categories. textscXMeCap achieves an average evaluation score of 75.85 for single-image memes and 66.32 for multi-image memes, outperforming the best baseline by 6.75% and 8.56%, respectively. This research not only establishes a new frontier in meme-related studies but also underscores the potential of machines in understanding and generating humor in a multi-modal setting.