# Cross-Modal Transfer from Memes to Videos: Addressing Data Scarcity in Hateful Video Detection

arXiv: https://arxiv.org/abs/2501.15438 · HTML: https://mlsanity.com/paper/2501.15438/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 26 Jan 2025.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Hate Speech Dataset from a White Supremacy Forum](https://mlsanity.com/paper/1809.04444/) (2018)
- [ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2206.13559/) (2022)
- [Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2110.00413/) (2021)
- [Detecting Online Hate Speech Using Context Aware Models](https://mlsanity.com/paper/1710.07395/) (2017)
- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection](https://mlsanity.com/paper/2308.08088/) (2023)
- [Evaluating GPT-3 Generated Explanations for Hateful Content Moderation](https://mlsanity.com/paper/2305.17680/) (2023)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [ToxiCloakCN: Evaluating Robustness of Offensive Language Detection in Chinese with Cloaking Perturbations](https://mlsanity.com/paper/2406.12223/) (2024)
- [DeepHate: Hate Speech Detection via Multi-Faceted Text Representations](https://mlsanity.com/paper/2103.11799/) (2021)
- [Temporal 3D ConvNets: New Architecture and Transfer Learning for Video Classification](https://mlsanity.com/paper/1711.08200/) (2017)
- [Recent Advances in Hate Speech Moderation: Multimodality and the Role of Large Models](https://mlsanity.com/paper/2401.16727/) (2024)
- [MultiHateClip: A Multilingual Benchmark Dataset for Hateful Video Detection on YouTube and Bilibili](https://mlsanity.com/paper/2408.03468/) (2024)
- [Modularized Networks for Few-shot Hateful Meme Detection](https://mlsanity.com/paper/2402.11845/) (2024)
- [Model-Agnostic Meta-Learning for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2303.02513/) (2023)
- [Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2311.15769/) (2023)
- [SGHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Singapore](https://mlsanity.com/paper/2405.01842/) (2024)

## Built on by (most cited first, 9 of 9)

- [MM-HSD: Multi-Modal Hate Speech Detection in Videos](https://mlsanity.com/paper/2508.20546/) (2025)
- [HateClipSeg: A Segment-Level Annotated Dataset for Fine-Grained Hate Video Detection](https://mlsanity.com/paper/2508.01712/) (2025)
- [Revealing Temporal Label Noise in Multimodal Hateful Video Classification](https://mlsanity.com/paper/2508.04900/) (2025)
- [Multimodal Hate Detection Using Dual-Stream Graph Neural Networks](https://mlsanity.com/paper/2509.13515/) (2025)
- [Reasoning-Aware Multimodal Fusion for Hateful Video Detection](https://mlsanity.com/paper/2512.02743/) (2025)
- [Training-Free and Interpretable Hateful Video Detection via Multi-stage Adversarial Reasoning](https://mlsanity.com/paper/2601.15115/) (2026)
- [Towards Training-free Multimodal Hate Localisation with Large Language Models](https://mlsanity.com/paper/2602.09637/) (2026)
- [QwenSafe: Multimodal Content Rating Description Identification via Preference-Aligned VLMs](https://mlsanity.com/paper/2605.20584/) (2026)
- [Decoding Multimodal Cues: Unveiling the Implicit Meaning Behind Hateful Videos](https://mlsanity.com/paper/2606.11953/) (2026)

## Abstract (arXiv)

Detecting hate speech in online content is essential to ensuring safer digital spaces. While significant progress has been made in text and meme modalities, video-based hate speech detection remains under-explored, hindered by a lack of annotated datasets and the high cost of video annotation. This gap is particularly problematic given the growing reliance on large models, which demand substantial amounts of training data. To address this challenge, we leverage meme datasets as both a substitution and an augmentation strategy for training hateful video detection models. Our approach introduces a human-assisted reannotation pipeline to align meme dataset labels with video datasets, ensuring consistency with minimal labeling effort. Using two state-of-the-art vision-language models, we demonstrate that meme data can substitute for video data in resource-scarce scenarios and augment video datasets to achieve further performance gains. Our results consistently outperform state-of-the-art benchmarks, showcasing the potential of cross-modal transfer learning for advancing hateful video detection. Dataset and code are available at https://github.com/Social-AI-Studio/CrossModalTransferLearning.