Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages
On the map: Knowledge distillation → Knowledge distillation for LLMs
Abstract (arXiv)
While impressive performance has been achieved on the task of Answer Sentence Selection (AS2) for English, the same does not hold for languages that lack large labeled datasets. In this work, we propose Cross-Lingual Knowledge Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2 models for low-resource languages in the tasks without the need of labeled data for the target language. To evaluate our method, we introduce 1) Xtr-WikiQA, a translation-based WikiQA dataset for 9 additional languages, and 2) TyDi-AS2, a multilingual AS2 dataset with over 70K questions spanning 8 typologically diverse languages. We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2 with multiple teachers, diverse monolingual and multilingual pretrained language models (PLMs) as students, and both monolingual and multilingual training. The results demonstrate that CLKD either outperforms or rivals even supervised fine-tuning with the same amount of labeled data and a combination of machine translation and the teacher model. Our method can potentially enable stronger AS2 models for low-resource languages, while TyDi-AS2 can serve as the largest multilingual AS2 dataset for further studies in the research community.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 35 of its 55 references that are in the map. First posted on arXiv 25 May 2023.
Builds on
35 of 55 references.
Closest ancestors · 2
Same direction, newest first
- 2021Learning Cross-Lingual IR from an English Retriever15 Dec
- 2021Distilling Linguistic Context for Language Model Compression17 Sep
Other references · 33
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
- 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
- 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
- 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
- 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
21 more
- 2019Multilingual is not enough: BERT for Finnishcited by 119
- 2021KLUE: Korean Language Understanding Evaluationcited by 114
- 2019Adaptation of Deep Bidirectional Multilingual Transformers for Russian Languagecited by 109
- 2019BERTje: A Dutch BERT Modelcited by 109
- 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
- 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
- 2019TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selectioncited by 74
- 2020PySBD: Pragmatic Sentence Boundary Disambiguationcited by 38
- 2020The Cascade Transformer: an Application for Efficient Answer Sentence Selectioncited by 20
- 2021Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answeringcited by 19
- 2021Answer Generation for Retrieval-based Question Answering Systemscited by 13
- 2021Joint Models for Answer Verification in Question Answering Systemscited by 13
- 2017Cross-lingual Distillation for Text Classificationcited by 9
- 2021Modeling Context in Answer Sentence Selection Systems on a Latency Budgetcited by 9
- 2022Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selectioncited by 9
- 2020Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teachercited by 7
- 2022Knowledge Transfer from Answer Ranking to Answer Generationcited by 6
- 2020A Study on Efficiency, Accuracy and Document Structure for Answer Sentence Selectioncited by 5
- 2021Multilingual Answer Sentence Reranking via Automatically Translated Datacited by 3
- 2022Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systemscited by 3
- 2021Reference-based Weak Supervision for Answer Sentence Selection using Web Datacited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2023Promoting Generalized Cross-lingual Question Answering in Few-resource Scenarios via Self-knowledge Distillation
- 2025CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answering
- 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation
- 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights
- 2022Multi-level Distillation of Semantic Knowledge for Pre-training Multilingual Language Model
- 2025Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models
- 2026Efficient Multilingual Dialogue Processing via Translation Pipelines and Distilled Language Models
- 2023Language Model Knowledge Distillation for Efficient Question Answering in Spanish