Culture-Aware Humorous Captioning: Multimodal Humor Generation across Cultural Contexts
On the map: Multimodal LLMs / vision-language models → Image editing and quality assessment
Abstract (arXiv)
Recent multimodal large language models have shown promising ability in generating humorous captions for images, yet they still lack stable control over explicit cultural context, making it difficult to jointly maintain image relevance, contextual appropriateness, and humor quality under a specified cultural background. To address this limitation, we introduce a new multimodal generation task, culture-aware humorous captioning, which requires a model to generate a humorous caption conditioned on both an input image and a target cultural context. Captions generated under different cultural contexts are not expected to share the same surface form, but should remain grounded in similar visual situations or humorous rationales.To support this task, we establish a six-dimensional evaluation framework covering image relevance, contextual fit, semantic richness, reasonableness, humor, and creativity. We further propose a staged alignment framework that first initializes the model with high-resource supervision under the Western cultural context, then performs multi-dimensional preference alignment via judge-based GRPO with a Degradation-aware Prototype Repulsion Constraint to mitigate reward hacking in open-ended generation, and finally adapts the model to the Eastern cultural context with a small amount of supervision. Experimental results show that our method achieves stronger overall performance under the proposed evaluation framework, with particularly large gains in contextual fit and a better balance between image relevance and humor under cultural constraints.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Image editing and quality assessment, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 75 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
64 of 75 references.
References · 64
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
38 more
- 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
- 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
- 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
- 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
- 2023Osprey: Pixel Understanding with Visual Instruction Tuningcited by 159
- 2024Investigating Cultural Alignment of Large Language Modelscited by 143
- 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
- 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
- 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
- 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
- 2023Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generationcited by 56
- 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 51
- 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 51
- 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
- 2024FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Modelcited by 45
- 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 41
- 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 34
- 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 31
- 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
- 2025Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimizationcited by 29
- 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
- 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
- 2025Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMscited by 26
- 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
- 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
- 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 19
- 2025Crowd Comparative Reasoning: Unlocking Comprehensive Evaluations for LLM-as-a-Judgecited by 19
- 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
- 2025SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learningcited by 16
- 2025CARE: Multilingual Human Preference Learning for Cultural Awarenesscited by 15
- 2024XMeCap: Meme Caption Generation with Sub-Image Adaptabilitycited by 14
- 2024ControlCap: Controllable Region-level Captioningcited by 13
- 2025RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Modelscited by 13
- 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
- 2023Learning Combinatorial Prompts for Universal Controllable Image Captioningcited by 12
- 2025Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMscited by 12
50 most cited of 64 All 64 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Computational Humor with Multimodal LLMs: Methods, Datasets, Evaluation, and Challenges
- 2026MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models
- 2024The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning
- 2024CIC: A Framework for Culturally-Aware Image Captioning
- 2026Beyond Translation: Cross-Cultural Meme Transcreation with Vision-Language Models
- 2026Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding
- 2025From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme
- 2025HumorPlanSearch: Structured Planning and HuCoT for Contextual AI Humor