# UniPSDA: Unsupervised Pseudo Semantic Data Augmentation for Zero-Shot Cross-Lingual Natural Language Understanding

arXiv: https://arxiv.org/abs/2406.16372 · HTML: https://mlsanity.com/paper/2406.16372/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 62 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration](https://mlsanity.com/paper/1705.09634/) (2017)
- [Multilingual Universal Sentence Encoder for Semantic Retrieval](https://mlsanity.com/paper/1907.04307/) (2019)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Detecting Visual Relationships with Deep Relational Networks](https://mlsanity.com/paper/1704.03114/) (2017)
- [CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP](https://mlsanity.com/paper/2006.06402/) (2020)
- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)
- [Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing](https://mlsanity.com/paper/1902.09492/) (2019)
- [Cross-lingual Retrieval for Iterative Self-Supervised Training](https://mlsanity.com/paper/2006.09526/) (2020)
- [GATE: Graph Attention Transformer Encoder for Cross-lingual Relation and Event Extraction](https://mlsanity.com/paper/2010.03009/) (2020)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering](https://mlsanity.com/paper/2010.12643/) (2020)
- [Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment](https://mlsanity.com/paper/2106.06381/) (2021)
- [Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction](https://mlsanity.com/paper/2203.08308/) (2022)
- [Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation](https://mlsanity.com/paper/2203.08757/) (2022)
- [PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN](https://mlsanity.com/paper/1708.01956/) (2017)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [XLM-K: Improving Cross-Lingual Language Model Pre-training with Multilingual Knowledge](https://mlsanity.com/paper/2109.12573/) (2021)
- [MultiBooked: A Corpus of Basque and Catalan Hotel Reviews Annotated for Aspect-level Sentiment Classification](https://mlsanity.com/paper/1803.08614/) (2018)
- [BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels](https://mlsanity.com/paper/1910.05040/) (2019)
- [A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model](https://mlsanity.com/paper/2201.10707/) (2022)
- [UXLA: A Robust Unsupervised Data Augmentation Framework for Zero-Resource Cross-Lingual NLP](https://mlsanity.com/paper/2004.13240/) (2020)
- [X-METRA-ADA: Cross-lingual Meta-Transfer Learning Adaptation to Natural Language Understanding and Question Answering](https://mlsanity.com/paper/2104.09696/) (2021)
- [Dual-Alignment Pre-training for Cross-lingual Sentence Embedding](https://mlsanity.com/paper/2305.09148/) (2023)
- [Multilingual Pre-training with Language and Task Adaptation for Multilingual Text Style Transfer](https://mlsanity.com/paper/2203.08552/) (2022)
- [Bilingual Sentiment Embeddings: Joint Projection of Sentiment Across Languages](https://mlsanity.com/paper/1805.09016/) (2018)
- [Cross-language Sentence Selection via Data Augmentation and Rationale Training](https://mlsanity.com/paper/2106.02293/) (2021)
- [Easy Adaptation to Mitigate Gender Bias in Multilingual Text Classification](https://mlsanity.com/paper/2204.05459/) (2022)

## Abstract (arXiv)

Cross-lingual representation learning transfers knowledge from resource-rich data to resource-scarce ones to improve the semantic understanding abilities of different languages. However, previous works rely on shallow unsupervised data generated by token surface matching, regardless of the global context-aware semantics of the surrounding text tokens. In this paper, we propose an Unsupervised Pseudo Semantic Data Augmentation (UniPSDA) mechanism for cross-lingual natural language understanding to enrich the training data without human interventions. Specifically, to retrieve the tokens with similar meanings for the semantic data augmentation across different languages, we propose a sequential clustering process in 3 stages: within a single language, across multiple languages of a language family, and across languages from multiple language families. Meanwhile, considering the multi-lingual knowledge infusion with context-aware semantics while alleviating computation burden, we directly replace the key constituents of the sentences with the above-learned multi-lingual family knowledge, viewed as pseudo-semantic. The infusion process is further optimized via three de-biasing techniques without introducing any neural parameters. Extensive experiments demonstrate that our model consistently improves the performance on general zero-shot cross-lingual natural language understanding tasks, including sequence classification, information extraction, and question answering.