mlsanityData to 14 Oct 2026

Multi-Modal Semantic Inconsistency Detection in Social Media News Posts

arXiv 2105.12855 ↗First posted 26 May 2021cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

As computer-generated content and deepfakes make steady improvements, semantic approaches to multimedia forensics will become more important. In this paper, we introduce a novel classification architecture for identifying semantic inconsistencies between video appearance and text caption in social media news posts. We develop a multi-modal fusion framework to identify mismatches between videos and captions in social media posts by leveraging an ensemble method based on textual analysis of the caption, automatic audio transcription, semantic video analysis, object detection, named entity consistency, and facial verification. To train and test our approach, we curate a new video-based dataset of 4,000 real-world Facebook news posts for analysis. Our multi-modal approach achieves 60.5% classification accuracy on random mismatches between caption and appearance, compared to accuracy below 50% for uni-modal models. Further ablation studies confirm the necessity of fusion across modalities for correctly identifying semantic inconsistencies.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 26 May 2021.

Builds on

34 of 55 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  10. 2017The Kinetics Human Action Video Datasetcited by 1,909
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
22 more
  1. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  2. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  3. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 847
  4. 2019Defending Against Neural Fake Newscited by 559
  5. 2017Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?cited by 551
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  8. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  9. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  10. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  11. 2021Multi-attentional Deepfake Detectioncited by 341
  12. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  13. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  14. 2018Multilevel Language and Vision Integration for Text-to-Clip Retrievalcited by 128
  15. 2020Emotions Don't Lie: An Audio-Visual Deepfake Detection Method Using Affective Cuescited by 111
  16. 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 107
  17. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  18. 2020DeepFake Detection by Analyzing Convolutional Tracescited by 81
  19. 2020Detecting Deep-Fake Videos from Appearance and Behaviorcited by 66
  20. 2019Detecting Photoshopped Faces by Scripting Photoshopcited by 55
  21. 2020Detecting Cross-Modal Inconsistency to Defend Against Neural Fake Newscited by 42
  22. 2021COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learningcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 1; 2026: 1.

10 papers.

  1. 2023COVID-VTS: Fact Extraction and Verification on Short Video Platformscited by 15
  2. 2023Combating Online Misinformation Videos: Characterization, Detection, and Future Directionscited by 14
  3. 2022An Overview of Recent Work in Media Forensics: Methods and Threatscited by 7
  4. 2023Not all Fake News is Written: A Dataset and Analysis of Misleading Video Headlinescited by 5
  5. 2026FactGuard: Agentic Video Misinformation Detection via Reinforcement Learningcited by 5
  6. 2022Misinformation Detection in Social Media Video Postscited by 4
  7. 2024Exposing Text-Image Inconsistency Using Diffusion Modelscited by 4
  8. 2024Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformationcited by 2
  9. 2024Interpretable Multimodal Out-of-context Detection with Soft Logic Regularizationcited by 2
  10. 2025Multimodal Fake News Video Explanation: Dataset, Analysis and Evaluationcited by 1