# Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understanding

arXiv: https://arxiv.org/abs/2606.03604 · HTML: https://mlsanity.com/paper/2606.03604/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 14 of 14)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training](https://mlsanity.com/paper/2505.22257/) (2025)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection](https://mlsanity.com/paper/2410.14731/) (2024)
- [Multi-source Semantic Graph-based Multimodal Sarcasm Explanation Generation](https://mlsanity.com/paper/2306.16650/) (2023)
- [Multi-Granular Multimodal Clue Fusion for Meme Understanding](https://mlsanity.com/paper/2503.12560/) (2025)
- [Meme-ingful Analysis: Enhanced Understanding of Cyberbullying in Memes Through Multimodal Explanations](https://mlsanity.com/paper/2401.09899/) (2024)
- [Incomplete Modality Disentangled Representation for Ophthalmic Disease Grading and Diagnosis](https://mlsanity.com/paper/2502.11724/) (2025)
- [Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation](https://mlsanity.com/paper/2505.11221/) (2025)
- [M-QUEST -- Meme Question-Understanding Evaluation on Semantics and Toxicity](https://mlsanity.com/paper/2603.03315/) (2026)
- [KID: Knowledge-Injected Dual-Head Learning for Knowledge-Grounded Harmful Meme Detection](https://mlsanity.com/paper/2601.21796/) (2026)
- [MORE-R1: Guiding LVLM for Multimodal Object-Entity Relation Extraction via Stepwise Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2603.09478/) (2026)
- [Rethinking Reinforcement Fine-Tuning in LVLM: Convergence, Reward Decomposition, and Generalization](https://mlsanity.com/paper/2604.19857/) (2026)

## Abstract (arXiv)

When asked what a meme or sarcastic post means, Large Vision Language Models (LVLMs) tend to describe what the image shows rather than what the author is trying to communicate. Standard instruction tuning entangles a post's literal content with its pragmatic meaning, letting surface-level details contaminate the final response. We reframe meme understanding as a problem of literal-pragmatic decomposition and propose Intent Projection, a framework that separates the two signals at the representation, output, and objective levels within a single LVLM backbone. At the representation level, an orthogonal projection module removes dominant unimodal directions from the fused image-text representation, retaining only the pragmatic residual, while a surface-real affect classifier anchors the decoder with a discrete tag that names the polarity gap. At the output level, the model externalizes a structured reasoning chain, and at the objective level a contrastive reward explicitly penalizes answers that restate the literal description. Across six multimodal benchmarks, Intent Projection consistently outperforms open-source baselines and narrows the gap to proprietary models, with the largest gains on high-divergence posts where literal collapse is most damaging.