# A Comparative Study of Light-weight Language Models for PII Masking and their Deployment for Real Conversational Texts

arXiv: https://arxiv.org/abs/2512.18608 · HTML: https://mlsanity.com/paper/2512.18608/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 24 references that are in the map.
- First posted on arXiv 21 Dec 2025.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Named Entity Recognition and Classification on Historical Documents: A Survey](https://mlsanity.com/paper/2109.11406/) (2021)
- [PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction](https://mlsanity.com/paper/2508.05545/) (2025)
- [Unmasking the Reality of PII Masking Models: Performance Gaps and the Call for Accountability](https://mlsanity.com/paper/2504.12308/) (2025)
- [Named Entity Recognition in Unstructured Medical Text Documents](https://mlsanity.com/paper/2110.15732/) (2021)
- [Life of PII -- A PII Obfuscation Transformer](https://mlsanity.com/paper/2305.09550/) (2023)
- [An Evaluation Study of Hybrid Methods for Multilingual PII Detection](https://mlsanity.com/paper/2510.07551/) (2025)

## Abstract (arXiv)

Automated masking of Personally Identifiable Information (PII) is critical for privacy-preserving conversational systems. While current frontier large language models demonstrate strong PII masking capabilities, concerns about data handling and computational costs motivate exploration of whether lightweight models can achieve comparable performance. We compare encoder-decoder and decoder-only architectures by fine-tuning T5-small and Mistral-Instruct-v0.3 on English datasets constructed from the AI4Privacy benchmark. We create different dataset variants to study label standardization and PII representation, covering 24 standardized PII categories and higher-granularity settings. Evaluation using entity-level and character-level metrics, type accuracy, and exact match shows that both lightweight models achieve performance comparable to frontier LLMs for PII masking tasks. Label normalization consistently improves performance across architectures. Mistral achieves higher F1 and recall with greater robustness across PII types but incurs significantly higher generation latency. T5, while less robust in conversational text, offers more controllable structured outputs and lower inference cost, motivating its use in a real-time Discord bot for real-world PII redaction. Evaluation on live messages reveals performance degradation under informal inputs. These results clarify trade-offs between accuracy, robustness, and computational efficiency, demonstrating that lightweight models can provide effective PII masking while addressing data handling concerns associated with frontier LLMs.