# Privacy-Preserving Online Content Moderation: A Federated Learning Use Case

arXiv: https://arxiv.org/abs/2209.11843 · HTML: https://mlsanity.com/paper/2209.11843/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Federated unlearning and poisoning, one of 12 directions in LLM safety and security.
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Placed in

- LLM safety and security / [Federated unlearning and poisoning](https://mlsanity.com/safety/federated-unlearning-and-poisoning/)

## Builds on (most cited first, 22 of 22)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Subsampled R'enyi Differential Privacy and Analytical Moments Accountant](https://mlsanity.com/paper/1808.00087/) (2018)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [R'enyi Differential Privacy of the Sampled Gaussian Mechanism](https://mlsanity.com/paper/1908.10530/) (2019)
- [Privacy and Robustness in Federated Learning: Attacks and Defenses](https://mlsanity.com/paper/2012.06337/) (2020)
- [LDP-Fed: Federated Learning with Local Differential Privacy](https://mlsanity.com/paper/2006.03637/) (2020)
- [Local and Central Differential Privacy for Robustness and Privacy in Federated Learning](https://mlsanity.com/paper/2009.03561/) (2020)
- [Federated Learning Of Out-Of-Vocabulary Words](https://mlsanity.com/paper/1903.10635/) (2019)
- [A Unified Deep Learning Architecture for Abuse Detection](https://mlsanity.com/paper/1802.00385/) (2018)
- [Experiments of Federated Learning for COVID-19 Chest X-ray Images](https://mlsanity.com/paper/2007.05592/) (2020)
- [Mean Birds: Detecting Aggression and Bullying on Twitter](https://mlsanity.com/paper/1702.06877/) (2017)
- [FLaaS: Federated Learning as a Service](https://mlsanity.com/paper/2011.09359/) (2020)
- [Disturbed YouTube for Kids: Characterizing and Detecting Inappropriate Videos Targeting Young Children](https://mlsanity.com/paper/1901.07046/) (2019)
- ["It is just a flu": Assessing the Effect of Watch History on YouTube's Pseudoscientific Video Recommendations](https://mlsanity.com/paper/2010.11638/) (2020)
- [Hate, Obscenity, and Insults: Measuring the Exposure of Children to Inappropriate Comments in YouTube](https://mlsanity.com/paper/2103.09050/) (2021)
- [Hierarchical Federated Learning with Privacy](https://mlsanity.com/paper/2206.05209/) (2022)

## Abstract (arXiv)

Users are daily exposed to a large volume of harmful content on various social network platforms. One solution is developing online moderation tools using Machine Learning techniques. However, the processing of user data by online platforms requires compliance with privacy policies. Federated Learning (FL) is an ML paradigm where the training is performed locally on the users' devices. Although the FL framework complies, in theory, with the GDPR policies, privacy leaks can still occur. For instance, an attacker accessing the final trained model can successfully perform unwanted inference of the data belonging to the users who participated in the training process. In this paper, we propose a privacy-preserving FL framework for online content moderation that incorporates Differential Privacy (DP). To demonstrate the feasibility of our approach, we focus on detecting harmful content on Twitter - but the overall concept can be generalized to other types of misbehavior. We simulate a text classifier - in FL fashion - which can detect tweets with harmful content. We show that the performance of the proposed FL framework can be close to the centralized approach - for both the DP and non-DP FL versions. Moreover, it has a high performance even if a small number of clients (each with a small number of data points) are available for the FL training. When reducing the number of clients (from 50 to 10) or the data points per client (from 1K to 0.1K), the classifier can still achieve ~81% AUC. Furthermore, we extend the evaluation to four other Twitter datasets that capture different types of user misbehavior and still obtain a promising performance (61% - 80% AUC). Finally, we explore the overhead on the users' devices during the FL training phase and show that the local training does not introduce excessive CPU utilization and memory consumption overhead.