# Is In-Context Learning Sufficient for Instruction Following in LLMs?

arXiv: https://arxiv.org/abs/2405.19874 · HTML: https://mlsanity.com/paper/2405.19874/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Data Engineering for Scaling Language Models to 128K Context](https://mlsanity.com/paper/2402.10171/) (2024)
- [RAIN: Your Language Models Can Align Themselves without Finetuning](https://mlsanity.com/paper/2309.07124/) (2023)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning](https://mlsanity.com/paper/2402.04833/) (2024)
- [In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning](https://mlsanity.com/paper/2308.04275/) (2023)
- [Exploring the Relationship between In-Context Learning and Instruction Tuning](https://mlsanity.com/paper/2311.10367/) (2023)

## Built on by (most cited first, 19 of 19)

- [Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements](https://mlsanity.com/paper/2410.08968/) (2024)
- [Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?](https://mlsanity.com/paper/2504.00509/) (2025)
- [Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models](https://mlsanity.com/paper/2411.08733/) (2024)
- [Revisiting the Superficial Alignment Hypothesis](https://mlsanity.com/paper/2410.03717/) (2024)
- [United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory](https://mlsanity.com/paper/2506.06843/) (2025)
- [CBF-LLM: Safe Control for LLM Alignment](https://mlsanity.com/paper/2408.15625/) (2024)
- [STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models](https://mlsanity.com/paper/2505.20645/) (2025)
- [MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?](https://mlsanity.com/paper/2502.09933/) (2025)
- [OptiSeq: Ordering Examples On-The-Fly for In-Context Learning](https://mlsanity.com/paper/2501.15030/) (2025)
- [In-context Learning vs. Instruction Tuning: The Case of Small and Multilingual Language Models](https://mlsanity.com/paper/2503.01611/) (2025)
- [Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory](https://mlsanity.com/paper/2608.07169/) (2026)
- [Is In-Context Learning Learning?](https://mlsanity.com/paper/2509.10414/) (2025)
- [DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction](https://mlsanity.com/paper/2510.09211/) (2025)
- [Learning-to-Context Slope: Evaluating In-Context Learning Effectiveness Beyond Performance Illusions](https://mlsanity.com/paper/2506.23146/) (2025)
- [TAdaRAG: Task Adaptive Retrieval-Augmented Generation via On-the-Fly Knowledge Graph Construction](https://mlsanity.com/paper/2511.12520/) (2025)
- [Control Barrier Function for Aligning Large Language Models](https://mlsanity.com/paper/2511.03121/) (2025)
- [Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?](https://mlsanity.com/paper/2603.28058/) (2026)
- [Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxies](https://mlsanity.com/paper/2605.09665/) (2026)
- [LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction](https://mlsanity.com/paper/2606.29407/) (2026)

## Abstract (arXiv)

In-context learning (ICL) allows LLMs to learn from examples without changing their weights: this is a particularly promising capability for long-context LLMs that can potentially learn from many examples. Recently, Lin et al. (2024) proposed URIAL, a method using only three in-context examples to align base LLMs, achieving non-trivial instruction following performance. In this work, we show that, while effective, ICL alignment with URIAL still underperforms compared to instruction fine-tuning on the established benchmark MT-Bench, especially with more capable base LLMs. We then uncover the most relevant elements for successful in-context alignment, finding the crucial role of the decoding parameters. Based on these insights, we show that the approach of URIAL can indeed be improved by adding high-quality, potentially carefully selected via greedy search, demonstrations in context, getting closer to the performance of instruct models. Finally, we provide the first, to our knowledge, systematic comparison of ICL and instruction fine-tuning (IFT) for instruction following in the low data regime, where ICL can be a viable alternative to IFT. Overall, our work advances the understanding of ICL as an alignment technique and its relationship to IFT. We provide our code at https://github.com/tml-epfl/icl-alignment.