# Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages

arXiv: https://arxiv.org/abs/2305.16302 · HTML: https://mlsanity.com/paper/2305.16302/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 25 May 2023.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [XOR QA: Cross-lingual Open-Retrieval Question Answering](https://mlsanity.com/paper/2010.11856/) (2020)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language](https://mlsanity.com/paper/1905.07213/) (2019)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [PySBD: Pragmatic Sentence Boundary Disambiguation](https://mlsanity.com/paper/2010.09657/) (2020)
- [Distilling Linguistic Context for Language Model Compression](https://mlsanity.com/paper/2109.08359/) (2021)
- [The Cascade Transformer: an Application for Efficient Answer Sentence Selection](https://mlsanity.com/paper/2005.02534/) (2020)
- [Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering](https://mlsanity.com/paper/2109.07009/) (2021)
- [Learning Cross-Lingual IR from an English Retriever](https://mlsanity.com/paper/2112.08185/) (2021)
- [Answer Generation for Retrieval-based Question Answering Systems](https://mlsanity.com/paper/2106.00955/) (2021)
- [Joint Models for Answer Verification in Question Answering Systems](https://mlsanity.com/paper/2107.04217/) (2021)
- [Cross-lingual Distillation for Text Classification](https://mlsanity.com/paper/1705.02073/) (2017)
- [Modeling Context in Answer Sentence Selection Systems on a Latency Budget](https://mlsanity.com/paper/2101.12093/) (2021)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher](https://mlsanity.com/paper/2010.02562/) (2020)

## Built on by (most cited first, 2 of 2)

- [The Signal is in the Steps: Local Scoring for Reasoning Data Selection](https://mlsanity.com/paper/2510.03988/) (2025)
- [Simple Unsupervised Knowledge Distillation With Space Similarity](https://mlsanity.com/paper/2409.13939/) (2024)

## Abstract (arXiv)

While impressive performance has been achieved on the task of Answer Sentence Selection (AS2) for English, the same does not hold for languages that lack large labeled datasets. In this work, we propose Cross-Lingual Knowledge Distillation (CLKD) from a strong English AS2 teacher as a method to train AS2 models for low-resource languages in the tasks without the need of labeled data for the target language. To evaluate our method, we introduce 1) Xtr-WikiQA, a translation-based WikiQA dataset for 9 additional languages, and 2) TyDi-AS2, a multilingual AS2 dataset with over 70K questions spanning 8 typologically diverse languages. We conduct extensive experiments on Xtr-WikiQA and TyDi-AS2 with multiple teachers, diverse monolingual and multilingual pretrained language models (PLMs) as students, and both monolingual and multilingual training. The results demonstrate that CLKD either outperforms or rivals even supervised fine-tuning with the same amount of labeled data and a combination of machine translation and the teacher model. Our method can potentially enable stronger AS2 models for low-resource languages, while TyDi-AS2 can serve as the largest multilingual AS2 dataset for further studies in the research community.