# Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts

arXiv: https://arxiv.org/abs/2204.04775 · HTML: https://mlsanity.com/paper/2204.04775/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 41 references that are in the map.
- First posted on arXiv 10 Apr 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis](https://mlsanity.com/paper/1706.03446/) (2017)
- [De-identification of Patient Notes with Recurrent Neural Networks](https://mlsanity.com/paper/1606.03475/) (2016)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [Few-Shot Named Entity Recognition: A Comprehensive Study](https://mlsanity.com/paper/2012.14978/) (2020)
- [Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework](https://mlsanity.com/paper/1910.04708/) (2019)
- [Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER](https://mlsanity.com/paper/1909.00153/) (2019)
- [Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1911.06161/) (2019)
- [UniTrans: Unifying Model Transfer and Data Transfer for Cross-Lingual Named Entity Recognition with Unlabeled Data](https://mlsanity.com/paper/2007.07683/) (2020)
- [AdvPicker: Effectively Leveraging Unlabeled Data via Adversarial Discriminator for Cross-Lingual NER](https://mlsanity.com/paper/2106.02300/) (2021)
- [Unsupervised Domain Adaptation on Reading Comprehension](https://mlsanity.com/paper/1911.06137/) (2019)
- [Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.09543/) (2021)
- [Adversarial Learning of Privacy-Preserving Text Representations for De-Identification of Medical Records](https://mlsanity.com/paper/1906.05000/) (2019)
- [Don't Use English Dev: On the Zero-Shot Cross-Lingual Evaluation of Contextual Embeddings](https://mlsanity.com/paper/2004.15001/) (2020)

## Built on by (most cited first, 5 of 5)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challenges](https://mlsanity.com/paper/2212.09660/) (2022)
- [Scalable multilingual PII annotation for responsible AI in LLMs](https://mlsanity.com/paper/2510.06250/) (2025)
- [An Evaluation Study of Hybrid Methods for Multilingual PII Detection](https://mlsanity.com/paper/2510.07551/) (2025)
- [Multilingual BERT language model for medical tasks: Evaluation on domain-specific adaptation and cross-linguality](https://mlsanity.com/paper/2510.27552/) (2025)

## Abstract (arXiv)

Despite the advances in digital healthcare systems offering curated structured knowledge, much of the critical information still lies in large volumes of unlabeled and unstructured clinical texts. These texts, which often contain protected health information (PHI), are exposed to information extraction tools for downstream applications, risking patient identification. Existing works in de-identification rely on using large-scale annotated corpora in English, which often are not suitable in real-world multilingual settings. Pre-trained language models (LM) have shown great potential for cross-lingual transfer in low-resource settings. In this work, we empirically show the few-shot cross-lingual transfer property of LMs for named entity recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke domain. We annotate a gold evaluation dataset to assess few-shot setting performance where we only use a few hundred labeled examples for training. Our model improves the zero-shot F1-score from 73.7% to 91.2% on the gold evaluation set when adapting Multilingual BERT (mBERT) (Devlin et al., 2019) from the MEDDOCAN (Marimon et al., 2019) corpus with our few-shot cross-lingual target corpus. When generalized to an out-of-sample test set, the best model achieves a human-evaluation F1-score of 97.2%.