# Bridging Pixels and Words: Mask-Aware Local Semantic Fusion for Multimodal Media Verification

arXiv: https://arxiv.org/abs/2603.26052 · HTML: https://mlsanity.com/paper/2603.26052/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 27 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [A Simple Framework for Open-Vocabulary Segmentation and Detection](https://mlsanity.com/paper/2303.08131/) (2023)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [Detecting and Grounding Multi-Modal Media Manipulation](https://mlsanity.com/paper/2304.02556/) (2023)
- [MDFEND: Multi-domain Fake News Detection](https://mlsanity.com/paper/2201.00987/) (2022)
- [Detecting and Grounding Multi-Modal Media Manipulation and Beyond](https://mlsanity.com/paper/2309.14203/) (2023)
- [Multimodal Fake News Detection via CLIP-Guided Learning](https://mlsanity.com/paper/2205.14304/) (2022)
- [FKA-Owl: Advancing Multimodal Fake News Detection through Knowledge-Augmented LVLMs](https://mlsanity.com/paper/2403.01988/) (2024)
- [Bootstrapping Multi-view Representations for Fake News Detection](https://mlsanity.com/paper/2206.05741/) (2022)
- [AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation](https://mlsanity.com/paper/2410.07164/) (2024)
- [Cross-modal Contrastive Learning for Multimodal Fake News Detection](https://mlsanity.com/paper/2302.14057/) (2023)
- [Pluralistic Aging Diffusion Autoencoder](https://mlsanity.com/paper/2303.11086/) (2023)
- [InstaStyle: Inversion Noise of a Stylized Image is Secretly a Style Adviser](https://mlsanity.com/paper/2311.15040/) (2023)
- [ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding](https://mlsanity.com/paper/2412.12718/) (2024)
- [Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation](https://mlsanity.com/paper/2309.09667/) (2023)
- [Frequency Spectrum is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector](https://mlsanity.com/paper/2312.11023/) (2023)
- [Modality Interactive Mixture-of-Experts for Fake News Detection](https://mlsanity.com/paper/2501.12431/) (2025)
- [Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding](https://mlsanity.com/paper/2309.12657/) (2023)
- [RaCMC: Residual-Aware Compensation Network with Multi-Granularity Constraints for Fake News Detection](https://mlsanity.com/paper/2412.18254/) (2024)

## Abstract (arXiv)

As multimodal misinformation becomes more sophisticated, its detection and grounding are crucial. However, current multimodal verification methods, relying on passive holistic fusion, struggle with sophisticated misinformation. Due to 'feature dilution,' global alignments tend to average out subtle local semantic inconsistencies, effectively masking the very conflicts they are designed to find. We introduce MaLSF (Mask-aware Local Semantic Fusion), a novel framework that shifts the paradigm to active, bidirectional verification, mimicking human cognitive cross-referencing. MaLSF utilizes mask-label pairs as semantic anchors to bridge pixels and words. Its core mechanism features two innovations: 1) a Bidirectional Cross-modal Verification (BCV) module that acts as an interrogator, using parallel query streams (Text-as-Query and Image-as-Query) to explicitly pinpoint conflicts; and 2) a Hierarchical Semantic Aggregation (HSA) module that intelligently aggregates these multi-granularity conflict signals for task-specific reasoning. In addition, to extract fine-grained mask-label pairs, we introduce a set of diverse mask-label pair extraction parsers. MaLSF achieves state-of-the-art performance on both the DGM4 and multimodal fake news detection tasks. Extensive ablation studies and visualization results further verify its effectiveness and interpretability.