# PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models

arXiv: https://arxiv.org/abs/2608.09772 · HTML: https://mlsanity.com/paper/2608.09772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality](https://mlsanity.com/paper/2306.14610/) (2023)
- [CREPE: Can Vision-Language Foundation Models Reason Compositionally?](https://mlsanity.com/paper/2212.07796/) (2022)
- [NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples](https://mlsanity.com/paper/2410.14669/) (2024)
- [Detecting Sarcasm in Multimodal Social Platforms](https://mlsanity.com/paper/1608.02289/) (2016)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQA](https://mlsanity.com/paper/2210.04692/) (2022)
- [The Hard Positive Truth about Vision-Language Compositionality](https://mlsanity.com/paper/2409.17958/) (2024)
- [Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning](https://mlsanity.com/paper/2402.17510/) (2024)
- [VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors](https://mlsanity.com/paper/2604.02486/) (2026)
- [Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding](https://mlsanity.com/paper/2509.22437/) (2025)
- [PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehension](https://mlsanity.com/paper/2412.11906/) (2024)
- [MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2510.23727/) (2025)
- [Schroedinger's Threshold: When the AUC doesn't predict Accuracy](https://mlsanity.com/paper/2404.03344/) (2024)
- [Multi-View Incongruity Learning for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2412.00756/) (2024)
- [Seeing to Generalize: How Visual Data Corrects Binding Shortcuts](https://mlsanity.com/paper/2602.15183/) (2026)
- [IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detection](https://mlsanity.com/paper/2505.16258/) (2025)
- [Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection](https://mlsanity.com/paper/2510.11852/) (2025)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial correlations, known as shortcut learning. This question is particularly important for multimodal sarcasm detection, where successful prediction depends on recognizing pragmatic incongruity rather than treating sarcasm as simple image-text mismatch. We introduce PragMatch, a controlled benchmark of 3,000 image-text pairs derived from MMSD2.0, including original sarcastic examples and constructed literal and hard-negative pairs. We identify influential shortcut cues through systematic masking and evaluate their impact through targeted injection experiments. Our results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships. Our findings reveal limitations in current LVLMs while PragMatch provides a systematic testbed for evaluating multimodal pragmatic reasoning beyond surface-level image-text alignment.