# Resources and Few-shot Learners for In-context Learning in Slavic Languages

arXiv: https://arxiv.org/abs/2304.01922 · HTML: https://mlsanity.com/paper/2304.01922/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 4 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [PAL: Program-aided Language Models](https://mlsanity.com/paper/2211.10435/) (2022)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation](https://mlsanity.com/paper/2104.08678/) (2021)
- [RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2010.15925/) (2020)
- [SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis](https://mlsanity.com/paper/1912.09723/) (2019)
- [POLYGLOT-NER: Massive Multilingual Named Entity Recognition](https://mlsanity.com/paper/1410.3791/) (2014)
- [Adaptor: Objective-Centric Adaptation Framework for Language Models](https://mlsanity.com/paper/2203.03989/) (2022)
- [CsFEVER and CTKFacts: Acquiring Czech data for fact verification](https://mlsanity.com/paper/2201.11115/) (2022)
- [A Survey on non-English Question Answering Dataset](https://mlsanity.com/paper/2112.13634/) (2021)
- [Political Advertising Dataset: the use case of the Polish 2020 Presidential Elections](https://mlsanity.com/paper/2006.10207/) (2020)

## Built on by (most cited first, 6 of 6)

- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism](https://mlsanity.com/paper/2412.17933/) (2024)
- [Concept-aware Data Construction Improves In-context Learning of Language Models](https://mlsanity.com/paper/2403.09703/) (2024)
- [Concept-aware Training Improves In-context Learning Ability of Language Models](https://mlsanity.com/paper/2305.13775/) (2023)
- [Enhancing BERT Fine-Tuning for Sentiment Analysis in Lower-Resourced Languages](https://mlsanity.com/paper/2512.01460/) (2025)
- [SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation](https://mlsanity.com/paper/2606.13647/) (2026)

## Abstract (arXiv)

Despite the rapid recent progress in creating accurate and compact in-context learners, most recent work focuses on in-context learning (ICL) for tasks in English. However, the ability to interact with users of languages outside English presents a great potential for broadening the applicability of language technologies to non-English speakers. In this work, we collect the infrastructure necessary for training and evaluation of ICL in a selection of Slavic languages: Czech, Polish, and Russian. We link a diverse set of datasets and cast these into a unified instructional format through a set of transformations and newly-crafted templates written purely in target languages. Using the newly-curated dataset, we evaluate a set of the most recent in-context learners and compare their results to the supervised baselines. Finally, we train, evaluate and publish a set of in-context learning models that we train on the collected resources and compare their performance to previous work. We find that ICL models tuned in English are also able to learn some tasks from non-English contexts, but multilingual instruction fine-tuning consistently improves the ICL ability. We also find that the massive multitask training can be outperformed by single-task training in the target language, uncovering the potential for specializing in-context learners to the language(s) of their application.