# From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension

arXiv: https://arxiv.org/abs/2112.04735 · HTML: https://mlsanity.com/paper/2112.04735/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 9 Dec 2021.

## Builds on (most cited first, 29 of 29)

- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [CoQA: A Conversational Question Answering Challenge](https://mlsanity.com/paper/1808.07042/) (2018)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Bidirectional Attention Flow for Machine Comprehension](https://mlsanity.com/paper/1611.01603/) (2016)
- [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://mlsanity.com/paper/2007.07834/) (2020)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension](https://mlsanity.com/paper/1804.09541/) (2018)
- [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019)
- [SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2108.06227/) (2021)
- [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://mlsanity.com/paper/1909.04849/) (2019)
- [Data Augmenting Contrastive Learning of Speech Representations in the Time Domain](https://mlsanity.com/paper/2007.00991/) (2020)
- [Momentum Contrastive Voxel-wise Representation Learning for Semi-supervised Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2105.07059/) (2021)
- [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021)
- [XLDA: Cross-Lingual Data Augmentation for Natural Language Inference and Question Answering](https://mlsanity.com/paper/1905.11471/) (2019)
- [Towards Data Distillation for End-to-end Spoken Conversational Question Answering](https://mlsanity.com/paper/2010.08923/) (2020)
- [Contextualized Attention-based Knowledge Transfer for Spoken Conversational Question Answering](https://mlsanity.com/paper/2010.11066/) (2020)
- [Adaptive Bi-directional Attention: Exploring Multi-Granularity Representations for Machine Reading Comprehension](https://mlsanity.com/paper/2012.10877/) (2020)
- [Multilingual Extractive Reading Comprehension by Runtime Machine Translation](https://mlsanity.com/paper/1809.03275/) (2018)
- [Cross-Lingual Machine Reading Comprehension](https://mlsanity.com/paper/1909.00361/) (2019)
- [Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension](https://mlsanity.com/paper/2004.14069/) (2020)
- [Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation](https://mlsanity.com/paper/2010.14271/) (2020)
- [CalibreNet: Calibration Networks for Multilingual Sequence Labeling](https://mlsanity.com/paper/2011.05723/) (2020)

## Built on by (most cited first, 10 of 10)

- [End-to-end Spoken Conversational Question Answering: Task, Dataset and Model](https://mlsanity.com/paper/2204.14272/) (2022)
- [Alleviating Over-smoothing for Unsupervised Sentence Representation](https://mlsanity.com/paper/2305.06154/) (2023)
- [Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling](https://mlsanity.com/paper/2204.05210/) (2022)
- [Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation](https://mlsanity.com/paper/2210.12902/) (2022)
- [Bridge the Gap between Language models and Tabular Understanding](https://mlsanity.com/paper/2302.09302/) (2023)
- [Natural Response Generation for Chinese Reading Comprehension](https://mlsanity.com/paper/2302.08817/) (2023)
- [Orca: A Few-shot Benchmark for Chinese Conversational Machine Reading Comprehension](https://mlsanity.com/paper/2302.13619/) (2023)
- [Promoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation](https://mlsanity.com/paper/2309.17134/) (2023)
- [An Understanding-Oriented Robust Machine Reading Comprehension Model](https://mlsanity.com/paper/2207.00187/) (2022)
- [Exploring and Exploiting Multi-Granularity Representations for Machine Reading Comprehension](https://mlsanity.com/paper/2208.08750/) (2022)

## Abstract (arXiv)

Cross-lingual Machine Reading Comprehension (xMRC) is challenging due to the lack of training data in low-resource languages. The recent approaches use training data only in a resource-rich language like English to fine-tune large-scale cross-lingual pre-trained language models. Due to the big difference between languages, a model fine-tuned only by a source language may not perform well for target languages. Interestingly, we observe that while the top-1 results predicted by the previous approaches may often fail to hit the ground-truth answers, the correct answers are often contained in the top-k predicted results. Based on this observation, we develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning (AA-CL) mechanism is developed to learn the fine difference between the accurate answer and other candidates. Our extensive experiments show that our model significantly outperforms a series of strong baselines on two cross-lingual MRC benchmark datasets.