# PersianAnonymizer: Evaluating LLM-Labeled Training for Efficient NER-based Anonymization in Persian

arXiv: https://arxiv.org/abs/2609.00958 · HTML: https://mlsanity.com/paper/2609.00958/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [ParsBERT: Transformer-based Model for Persian Language Understanding](https://mlsanity.com/paper/2005.12515/) (2020)
- [LLMaAA: Making Large Language Models as Active Annotators](https://mlsanity.com/paper/2310.19596/) (2023)
- [Understanding and Mitigating Language Confusion in LLMs](https://mlsanity.com/paper/2406.20052/) (2024)
- [PHICON: Improving Generalization of Clinical Text De-identification Models via Data Augmentation](https://mlsanity.com/paper/2010.05143/) (2020)
- [Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT](https://mlsanity.com/paper/2404.02403/) (2024)
- [Beheshti-NER: Persian Named Entity Recognition Using BERT](https://mlsanity.com/paper/2003.08875/) (2020)
- [PEYMA: A Tagged Corpus for Persian Named Entities](https://mlsanity.com/paper/1801.09936/) (2018)
- [Attack Named Entity Recognition by Entity Boundary Interference](https://mlsanity.com/paper/2305.05253/) (2023)
- [LLMs-in-the-Loop Part 2: Expert Small AI Models for Anonymization and De-identification of PHI Across Multiple Languages](https://mlsanity.com/paper/2412.10918/) (2024)
- [Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization](https://mlsanity.com/paper/2501.04858/) (2025)

## Abstract (arXiv)

We target practical anonymization of Persian customer chats by training a compact NER model from LLM-labeled supervision and selecting the best labeler for deployment. We compare three instruction-tuned LLMs: DeepSeek-V3-0324, GPT-OSS-120B, and Qwen3-235B-A22B-Instruct-2507, to produce span annotations under a shared JSON protocol, yielding four corpora (OSS_ZeroShot, Qwen_ZeroShot, Qwen_FewShot, DeepSeek_FewShot). A MatinaRoberta-based token-classifier is trained per corpus and evaluated with token-level Precision/Recall/F1 (overall and per-class). We also report Label Coverage Recall (LCR), the proportion of gold non-O tokens predicted as non-O, and quantify cross-labeler behavior via a token-level Venn on test annotations. Finally, we contrast test-set annotation latency of the LLMs on H200 nodes with the trained NER's test-time labeling on a single RTX 3090. Results show that supervision from OSS_ZeroShot yields the strongest macro-F1 and LCR, while the resulting NER labels an entire 40K-message test set in approximately 2 minutes on one consumer GPU. This establishes a practical path to high-quality, low-cost anonymization for Persian industrial data.