# Large Language Models in the Abuse Detection Pipeline

arXiv: https://arxiv.org/abs/2604.00323 · HTML: https://mlsanity.com/paper/2604.00323/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 74 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations](https://mlsanity.com/paper/2411.10414/) (2024)
- [Stylometric Detection of AI-Generated Text in Twitter Timelines](https://mlsanity.com/paper/2303.03697/) (2023)
- [Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025](https://mlsanity.com/paper/2504.09737/) (2025)
- [Evaluating GPT-3 Generated Explanations for Hateful Content Moderation](https://mlsanity.com/paper/2305.17680/) (2023)
- [AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration](https://mlsanity.com/paper/2503.15754/) (2025)
- [HARE: Explainable Hate Speech Detection with Step-by-Step Reasoning](https://mlsanity.com/paper/2311.00321/) (2023)
- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM](https://mlsanity.com/paper/2508.05775/) (2025)
- [Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implications](https://mlsanity.com/paper/2602.16201/) (2026)
- [Large Language Models as 'Hidden Persuaders': Fake Product Reviews are Indistinguishable to Humans and Machines](https://mlsanity.com/paper/2506.13313/) (2025)
- [Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning](https://mlsanity.com/paper/2602.21103/) (2026)
- [Detecting Instruction Fine-tuning Attacks using Influence Function](https://mlsanity.com/paper/2504.09026/) (2025)
- [BotHash: Efficient and Training-Free Bot Detection Through Approximate Nearest Neighbor](https://mlsanity.com/paper/2506.20503/) (2025)
- [Detecting LLM-Generated Spam Reviews by Integrating Language Model Embeddings and Graph Neural Network](https://mlsanity.com/paper/2510.01801/) (2025)
- [Increasing AI Explainability by LLM Driven Standard Processes](https://mlsanity.com/paper/2511.07083/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Silent Vote: Improving Zero-Shot LLM Reliability by Aggregating Semantic Neighborhoods](https://mlsanity.com/paper/2605.09739/) (2026)

## Abstract (arXiv)

Online abuse has grown increasingly complex, spanning toxic language, harassment, manipulation, and fraudulent behavior. Traditional machine-learning approaches dependent on static classifiers and labor-intensive labeling struggle to keep pace with evolving threat patterns and nuanced policy requirements. Large Language Models introduce new capabilities for contextual reasoning, policy interpretation, explanation generation, and cross-modal understanding, enabling them to support multiple stages of modern safety systems. This survey provides a lifecycle-oriented analysis of how LLMs are being integrated into the Abuse Detection Lifecycle (ADL), which we define across four stages: (I) Label & Feature Generation, (II) Detection, (III) Review & Appeals, and (IV) Auditing & Governance. For each stage, we synthesize emerging research and industry practices, highlight architectural considerations for production deployment, and examine the strengths and limitations of LLM-driven approaches. We conclude by outlining key challenges including latency, cost-efficiency, determinism, adversarial robustness, and fairness and discuss future research directions needed to operationalize LLMs as reliable, accountable components of large-scale abuse-detection and governance systems.