mlsanityData to 7 Oct 2026

In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models

arXiv 2503.01611 ↗First posted 3 Mar 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Instruction following is a critical ability for Large Language Models to be used directly by humans. This often requires supervised fine-tuning on curated instruction datasets, sometimes complemented with an alignment step. However, in multilingual scenarios, obtaining high-quality data for these stages remains challenging, motivating the exploration of In-Context Learning (ICL) as a possible alternative. In this work, we study whether ICL can serve as a substitute for Instruction Tuning in multilingual language models, while also examining how the comparison changes with model scale. Our results indicate that a gap remains between ICL and Instruction Tuning, motivating further research to reduce it.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 31 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

20 of 31 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  10. 2023LIMA: Less Is More for Alignmentcited by 1,080
  11. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  12. 2022Democratizing Neural Machine Translation with OPUS-MTcited by 57
8 more
  1. 2023Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languagescited by 52
  2. 2023FinGPT: Large Generative Models for a Small Languagecited by 49
  3. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  4. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  5. 2023In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuningcited by 19
  6. 2024Is In-Context Learning Sufficient for Instruction Following in LLMs?cited by 19
  7. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
  8. 2024BgGPT 1.0: Extending English-centric LLMs to other languagescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selectioncited by 7
  2. 2025BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examplescited by 0
  3. 2026Learn-To-Learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-Gated LLMcited by 0