mlsanityData to 7 Oct 2026

PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Models

arXiv 2608.09772 ↗First posted 10 Aug 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated strong performance on multimodal benchmarks, yet it remains unclear whether they genuinely reason about relationships between images and text or rely on superficial correlations, known as shortcut learning. This question is particularly important for multimodal sarcasm detection, where successful prediction depends on recognizing pragmatic incongruity rather than treating sarcasm as simple image-text mismatch. We introduce PragMatch, a controlled benchmark of 3,000 image-text pairs derived from MMSD2.0, including original sarcastic examples and constructed literal and hard-negative pairs. We identify influential shortcut cues through systematic masking and evaluate their impact through targeted injection experiments. Our results show that LVLM predictions are sensitive to lexical, OCR-derived and stylistic cues, with injected surface signals causing substantial changes in model predictions despite unchanged underlying image-text relationships. Our findings reveal limitations in current LVLMs while PragMatch provides a systematic testbed for evaluating multimodal pragmatic reasoning beyond surface-level image-text alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 29 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

20 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Evaluating Open-Source Vision-Language Models for Multimodal Sarcasm Detection13 Oct
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  3. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  4. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  5. 2016Detecting Sarcasm in Multimodal Social Platformscited by 43
  6. 2023MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection Systemcited by 31
  7. 2022Language Prior Is Not the Only Shortcut: A Benchmark for Shortcut Learning in VQAcited by 30
  8. 2024The Hard Positive Truth about Vision-Language Compositionalitycited by 21
  9. 2024Demonstrating and Reducing Shortcuts in Vision-Language Representation Learningcited by 14
  10. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 12
  11. 2025Chimera: Diagnosing Shortcut Learning in Visual-Language Understandingcited by 6
  12. 2024PunchBench: Benchmarking MLLMs in Multimodal Punchline Comprehensioncited by 5
5 more
  1. 2025MUStReason: A Benchmark for Diagnosing Pragmatic Reasoning in Video-LMs for Multimodal Sarcasm Detectioncited by 5
  2. 2024Schroedinger's Threshold: When the AUC doesn't predict Accuracycited by 4
  3. 2024Multi-View Incongruity Learning for Multimodal Sarcasm Detectioncited by 4
  4. 2026Seeing to Generalize: How Visual Data Corrects Binding Shortcutscited by 4
  5. 2025IRONIC: Coherence-Aware Reasoning Chains for Multi-Modal Sarcasm Detectioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing Sarcasm Through Different Eyes: Analyzing Multimodal Sarcasm Perception in Large Vision-Language Models
  2. 2025Think Twice Before You Judge: Mixture of Dual Reasoning Experts for Multimodal Sarcasm Detection
  3. 2025Can Large Vision-Language Models Understand Multimodal Sarcasm?
  4. 2025MMSD3.0: A Multi-Image Benchmark for Real-World Multimodal Sarcasm Detection
  5. 2026GDCNet: Generative Discrepancy Comparison Network for Multimodal Sarcasm Detection
  6. 2026URMF: Uncertainty-aware Robust Multimodal Fusion for Multimodal Sarcasm Detection
  7. 2024CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models
  8. 2025Evaluating Multimodal Large Language Models on Spoken Sarcasm Understanding