# Embedding-based Retrieval in Multimodal Content Moderation

arXiv: https://arxiv.org/abs/2507.01066 · HTML: https://mlsanity.com/paper/2507.01066/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [Why are Visually-Grounded Language Models Bad at Image Classification?](https://mlsanity.com/paper/2405.18415/) (2024)
- [Embedding-based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2106.09297/) (2021)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning](https://mlsanity.com/paper/2310.03400/) (2023)
- [ICM-Assistant: Instruction-tuning Multimodal Large Language Models for Rule-based Explainable Image Content Moderation](https://mlsanity.com/paper/2412.18216/) (2024)
- [Rethinking Multimodal Content Moderation from an Asymmetric Angle with Mixed-modality](https://mlsanity.com/paper/2305.10547/) (2023)
- [Enhancing Relevance of Embedding-based Retrieval at Walmart](https://mlsanity.com/paper/2408.04884/) (2024)
- [Multilingual Content Moderation: A Case Study on Reddit](https://mlsanity.com/paper/2302.09618/) (2023)
- [CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification](https://mlsanity.com/paper/2410.03038/) (2024)
- [Embedding based retrieval for long tail search queries in ecommerce](https://mlsanity.com/paper/2505.01946/) (2025)

## Built on by (most cited first, 4 of 4)

- [Reasoning-Enhanced Domain-Adaptive Pretraining of Multimodal Large Language Models for Short Video Content Governance](https://mlsanity.com/paper/2509.21486/) (2025)
- [MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models](https://mlsanity.com/paper/2608.22236/) (2026)
- [Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration](https://mlsanity.com/paper/2605.11591/) (2026)
- [UNIVID: Unified Vision-Language Model for Video Moderation](https://mlsanity.com/paper/2606.05748/) (2026)

## Abstract (arXiv)

Video understanding plays a fundamental role for content moderation on short video platforms, enabling the detection of inappropriate content. While classification remains the dominant approach for content moderation, it often struggles in scenarios requiring rapid and cost-efficient responses, such as trend adaptation and urgent escalations. To address this issue, we introduce an Embedding-Based Retrieval (EBR) method designed to complement traditional classification approaches. We first leverage a Supervised Contrastive Learning (SCL) framework to train a suite of foundation embedding models, including both single-modal and multi-modal architectures. Our models demonstrate superior performance over established contrastive learning methods such as CLIP and MoCo. Building on these embedding models, we design and implement the embedding-based retrieval system that integrates embedding generation and video retrieval to enable efficient and effective trend handling. Comprehensive offline experiments on 25 diverse emerging trends show that EBR improves ROC-AUC from 0.85 to 0.99 and PR-AUC from 0.35 to 0.95. Further online experiments reveal that EBR increases action rates by 10.32% and reduces operational costs by over 80%, while also enhancing interpretability and flexibility compared to classification-based solutions.