mlsanityData to 14 Oct 2026

Is In-Context Learning Sufficient for Instruction Following in LLMs?

arXiv 2405.19874 ↗First posted 30 May 2024cs.CL cs.AI cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICL alignment with URIAL still underperforms compared to instruction fine-tuning on the established benchmark MT-Bench, especially with more capable base LLMs. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding high-quality, potentially carefully selected via greedy search, demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime, where ICL can be a viable alternative to IFT. Overall, our work advances the understanding of ICL as an alignment technique and its relationship to IFT. We provide our code at https://github.com/tml-epfl/icl-alignment.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

28 of 39 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  12. 2024Mixtral of Expertscited by 1,823
16 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  2. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2023LIMA: Less Is More for Alignmentcited by 1,100
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  6. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  7. 2023The False Promise of Imitating Proprietary LLMscited by 263
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  9. 2024Many-Shot In-Context Learningcited by 206
  10. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  11. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  12. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  13. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  14. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  15. 2023In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuningcited by 19
  16. 2023Exploring the Relationship between In-Context Learning and Instruction Tuningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 11; 2026: 4.

19 papers.

Sort:

By area:

  1. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 36
  2. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  3. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Modelscited by 13
  4. 2024Revisiting the Superficial Alignment Hypothesiscited by 12
  5. 2025United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theorycited by 10
  6. 2024CBF-LLM: Safe Control for LLM Alignmentcited by 9
  7. 2025STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Modelscited by 8
  8. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7
  9. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learningcited by 6
  10. 2025In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Modelscited by 3
  11. 2026Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memorycited by 3
  12. 2025Is In-Context Learning Learning?cited by 2
7 more
  1. 2025DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correctioncited by 2
  2. 2025Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusionscited by 1
  3. 2025TAdaRAG: Task Adaptive Retrieval-Augmented Generation via On-the-Fly Knowledge Graph Constructioncited by 1
  4. 2025Control Barrier Function for Aligning Large Language Modelscited by 0
  5. 2026Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?cited by 0
  6. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0
  7. 2026LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extractioncited by 0