# An Expanded Synthetic Conversation Dataset for Multi-Turn Smishing Detection

arXiv: https://arxiv.org/abs/2606.06879 · HTML: https://mlsanity.com/paper/2606.06879/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Reframing Instructional Prompts to GPTk's Language](https://mlsanity.com/paper/2109.07830/) (2021)
- [Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts](https://mlsanity.com/paper/2209.12711/) (2022)
- [Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detection](https://mlsanity.com/paper/2503.15552/) (2025)
- [An analysis of scam baiting calls: Identifying and extracting scam stages and scripts](https://mlsanity.com/paper/2307.01965/) (2023)
- [Bot Wars Evolved: Orchestrating Competing LLMs in a Counterstrike Against Phone Scams](https://mlsanity.com/paper/2503.07036/) (2025)
- [Victim as a Service: Designing a System for Engaging with Interactive Scammers](https://mlsanity.com/paper/2510.23927/) (2025)
- [Bridging the Protection Gap: Innovative Approaches to Shield Older Adults from AI-Enhanced Scams](https://mlsanity.com/paper/2409.18249/) (2024)
- [A Synthetic Conversational Smishing Dataset for Social Engineering Detection](https://mlsanity.com/paper/2604.11752/) (2026)

## Abstract (arXiv)

Our prior work introduced COVA, a synthetically generated multi-turn conversational smishing dataset of 3,201 labeled conversations, establishing baseline detection benchmarks across eight models. While XGBoost with TF-IDF features achieved the best performance, with 72.5% accuracy and 0.691 macro F1, transformer models underperformed, which was attributed to input truncation and insufficient training data. We present COVA-X, an expanded dataset of 10,985 conversations spanning eight elder-targeted scam categories, produced by an improved generation pipeline addressing contamination, label mismatch, stage-direction bleed, and prompt-design failures from the first iteration. Retraining all classifiers on the expanded dataset yields the central finding of this work: Longformer now surpasses XGBoost on all evaluation metrics, achieving 79.71% accuracy and 0.7786 macro F1 compared with 78.43% and 0.7563 for XGBoost. This directly confirms that transformer models require larger conversational corpora to realize their contextual advantages. We additionally document a quality life-cycle including a 12.7× improvement in label correction rate, from 49.8% to 3.9%, an architectural intervention reducing virtual-kidnapping artifact rates from 67.1% to 46.5%, and a per-scam-type outcome analysis showing that scam categories modulate results in mechanism-consistent ways. A pre/post-cleanup sensitivity analysis confirms that dataset refinement recovers genuine label-relevant signal across all three classifier architectures.