# A Human-ML Collaboration Framework for Improving Video Content Reviews

arXiv: https://arxiv.org/abs/2210.09500 · HTML: https://mlsanity.com/paper/2210.09500/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [YouTube-8M: A Large-Scale Video Classification Benchmark](https://mlsanity.com/paper/1609.08675/) (2016)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection](https://mlsanity.com/paper/1811.07901/) (2018)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants](https://mlsanity.com/paper/2112.09062/) (2021)
- [The Use of Machine Learning Algorithms in Recommender Systems: A Systematic Review](https://mlsanity.com/paper/1511.05263/) (2015)
- [Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience](https://mlsanity.com/paper/2001.09219/) (2020)
- [One-bit Supervision for Image Classification](https://mlsanity.com/paper/2009.06168/) (2020)
- [The MeLa BitChute Dataset](https://mlsanity.com/paper/2202.05364/) (2022)

## Built on by (most cited first, 1 of 1)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)

## Abstract (arXiv)

We deal with the problem of localized in-video taxonomic human annotation in the video content moderation domain, where the goal is to identify video segments that violate granular policies, e.g., community guidelines on an online video platform. High quality human labeling is critical for enforcement in content moderation. This is challenging due to the problem of information overload - raters need to apply a large taxonomy of granular policy violations with ambiguous definitions, within a limited review duration to relatively long videos. Our key contribution is a novel human-machine learning (ML) collaboration framework aimed at maximizing the quality and efficiency of human decisions in this setting - human labels are used to train segment-level models, the predictions of which are displayed as "hints" to human raters, indicating probable regions of the video with specific policy violations. The human verified/corrected segment labels can help refine the model further, hence creating a human-ML positive feedback loop. Experiments show improved human video moderation decision quality, and efficiency through more granular annotations submitted within a similar review duration, which enable a 5-8% AUC improvement in the hint generation models.