# Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2204.00996 · HTML: https://mlsanity.com/paper/2204.00996/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 27 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Finding Universal Grammatical Relations in Multilingual BERT](https://mlsanity.com/paper/2005.04511/) (2020)
- [A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations](https://mlsanity.com/paper/1904.01173/) (2019)
- [StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing](https://mlsanity.com/paper/1806.07832/) (2018)
- [Syntax-Infused Variational Autoencoder for Text Generation](https://mlsanity.com/paper/1906.02181/) (2019)
- [Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model](https://mlsanity.com/paper/1909.09587/) (2019)
- [BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels](https://mlsanity.com/paper/1910.05040/) (2019)
- [Cross-Lingual Machine Reading Comprehension](https://mlsanity.com/paper/1909.00361/) (2019)
- [Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension](https://mlsanity.com/paper/2004.14069/) (2020)
- [Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation](https://mlsanity.com/paper/2010.14271/) (2020)
- [CalibreNet: Calibration Networks for Multilingual Sequence Labeling](https://mlsanity.com/paper/2011.05723/) (2020)
- [Improving Low-resource Reading Comprehension via Cross-lingual Transposition Rethinking](https://mlsanity.com/paper/2107.05002/) (2021)
- [Improving Cross-Lingual Reading Comprehension with Self-Training](https://mlsanity.com/paper/2105.03627/) (2021)

## Built on by (most cited first, 6 of 6)

- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Precedent-Enhanced Legal Judgment Prediction with LLM and Domain-Model Collaboration](https://mlsanity.com/paper/2310.09241/) (2023)
- [UDApter -- Efficient Domain Adaptation Using Adapters](https://mlsanity.com/paper/2302.03194/) (2023)
- [Cross-Lingual Question Answering over Knowledge Base as Reading Comprehension](https://mlsanity.com/paper/2302.13241/) (2023)
- [Towards Zero-Shot Functional Compositionality of Language Models](https://mlsanity.com/paper/2303.03103/) (2023)

## Abstract (arXiv)

Multilingual pre-trained models are able to zero-shot transfer knowledge from rich-resource to low-resource languages in machine reading comprehension (MRC). However, inherent linguistic discrepancies in different languages could make answer spans predicted by zero-shot transfer violate syntactic constraints of the target language. In this paper, we propose a novel multilingual MRC framework equipped with a Siamese Semantic Disentanglement Model (SSDM) to disassociate semantics from syntax in representations learned by multilingual pre-trained models. To explicitly transfer only semantic knowledge to the target language, we propose two groups of losses tailored for semantic and syntactic encoding and disentanglement. Experimental results on three multilingual MRC datasets (i.e., XQuAD, MLQA, and TyDi QA) demonstrate the effectiveness of our proposed approach over models based on mBERT and XLM-100. Code is available at:https://github.com/wulinjuan/SSDM_MRC.