# AI-in-the-Loop: Privacy Preserving Real-Time Scam Detection and Conversational Scambaiting by Leveraging LLMs and Federated Learning

arXiv: https://arxiv.org/abs/2509.05362 · HTML: https://mlsanity.com/paper/2509.05362/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 93 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 30 of 37)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [A Diversity-Promoting Objective Function for Neural Conversation Models](https://mlsanity.com/paper/1510.03055/) (2015)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Deep Leakage from Gradients](https://mlsanity.com/paper/1906.08935/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Opacus: User-Friendly Differential Privacy Library in PyTorch](https://mlsanity.com/paper/2109.12298/) (2021)
- [iDLG: Improved Deep Leakage from Gradients](https://mlsanity.com/paper/2001.02610/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [Improving Generalization in Federated Learning by Seeking Flat Minima](https://mlsanity.com/paper/2203.11834/) (2022)
- [Federated Learning: Opportunities and Challenges](https://mlsanity.com/paper/2101.05428/) (2021)
- [GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems](https://mlsanity.com/paper/2010.03994/) (2020)
- [Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks](https://mlsanity.com/paper/2310.13800/) (2023)
- [Bounding Training Data Reconstruction in DP-SGD](https://mlsanity.com/paper/2302.07225/) (2023)
- [Predictive Engagement: An Efficient Metric For Automatic Evaluation of Open-Domain Dialogue Systems](https://mlsanity.com/paper/1911.01456/) (2019)
- [Stop using the elbow criterion for k-means and how to choose the number of clusters instead](https://mlsanity.com/paper/2212.12189/) (2022)
- [On Calibration of LLM-based Guard Models for Reliable Content Moderation](https://mlsanity.com/paper/2410.10414/) (2024)
- [URLTran: Improving Phishing URL Detection Using Transformers](https://mlsanity.com/paper/2106.05256/) (2021)

## Built on by (most cited first, 1 of 1)

- [Anatomy of a Scam Call: What 10,000 real scam and spam calls reveal about how phone scammers operate](https://mlsanity.com/paper/2608.24127/) (2026)

## Abstract (arXiv)

Scams exploiting real-time social engineering -- such as phishing, impersonation, and phone fraud -- remain a persistent and evolving threat across digital platforms. Existing defenses are largely reactive, offering limited protection during active interactions. We propose a privacy-preserving, AI-in-the-loop framework that proactively detects and disrupts scam conversations in real time. The system combines instruction-tuned artificial intelligence with a safety-aware utility function that balances engagement with harm minimization, and employs federated learning to enable continual model updates without raw data sharing. Experimental evaluations show that the system produces fluent and engaging responses (perplexity as low as 22.3, engagement ≈0.80), while human studies confirm significant gains in realism, safety, and effectiveness over strong baselines. In federated settings, models trained with FedAvg sustain up to 30 rounds while preserving high engagement (≈0.80), strong relevance (≈0.74), and low PII leakage (≤0.0085). Even with differential privacy, novelty and safety remain stable, indicating that robust privacy can be achieved without sacrificing performance. The evaluation of guard models (LlamaGuard, LlamaGuard2/3, MD-Judge) shows a straightforward pattern: stricter moderation settings reduce the chance of exposing personal information, but they also limit how much the model engages in conversation. In contrast, more relaxed settings allow longer and richer interactions, which improve scam detection, but at the cost of higher privacy risk. To our knowledge, this is the first framework to unify real-time scam-baiting, federated privacy preservation, and calibrated safety moderation into a proactive defense paradigm.