# Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning

arXiv: https://arxiv.org/abs/2401.06469 · HTML: https://mlsanity.com/paper/2401.06469/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Transformers as Algorithms: Generalization and Stability in In-context Learning](https://mlsanity.com/paper/2301.07067/) (2023)
- [Parallel Context Windows for Large Language Models](https://mlsanity.com/paper/2212.10947/) (2022)
- [The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention](https://mlsanity.com/paper/2202.05798/) (2022)
- [Towards Understanding How Transformers Learn In-context Through a Representation Learning Lens](https://mlsanity.com/paper/2310.13220/) (2023)

## Built on by (most cited first, 20 of 20)

- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake News](https://mlsanity.com/paper/2403.09498/) (2024)
- [StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason](https://mlsanity.com/paper/2507.02841/) (2025)
- [Finding Visual Task Vectors](https://mlsanity.com/paper/2404.05729/) (2024)
- [Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2410.16983/) (2024)
- [Mixture of In-Context Experts Enhance LLMs' Long Context Awareness](https://mlsanity.com/paper/2406.19598/) (2024)
- [More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives](https://mlsanity.com/paper/2501.04070/) (2025)
- [Rethinking Invariance in In-context Learning](https://mlsanity.com/paper/2505.04994/) (2025)
- [OptiSeq: Ordering Examples On-The-Fly for In-Context Learning](https://mlsanity.com/paper/2501.15030/) (2025)
- [Language Models "Grok" to Copy](https://mlsanity.com/paper/2409.09281/) (2024)
- [Adaptive Task Vectors for Large Language Models](https://mlsanity.com/paper/2506.03426/) (2025)
- [Hearing the Order: Investigating Position Bias in Large Audio-Language Models](https://mlsanity.com/paper/2510.00628/) (2025)
- [ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedback](https://mlsanity.com/paper/2503.21332/) (2025)
- [Disentangling Latent Shifts of In-Context Learning with Weak Supervision](https://mlsanity.com/paper/2410.01508/) (2024)
- [MILE: A Mutation Testing Framework of In-Context Learning Systems](https://mlsanity.com/paper/2409.04831/) (2024)
- [PLR: Plackett-Luce for Reordering In-Context Learning Examples](https://mlsanity.com/paper/2603.21373/) (2026)
- [Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection](https://mlsanity.com/paper/2605.18512/) (2026)
- [Distributed In-Context Learning under Non-IID Among Clients](https://mlsanity.com/paper/2408.00144/) (2024)
- [Self-Improving In-Context Learning](https://mlsanity.com/paper/2605.23180/) (2026)
- [Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptations](https://mlsanity.com/paper/2610.00928/) (2026)

## Abstract (arXiv)

In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to the development of Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for ICL. Differing from the standard N-shot learning approach, Batch-ICL employs N separate 1-shot forward computations and aggregates the resulting meta-gradients. These aggregated meta-gradients are then applied to the forward computation of a zero-shot query to generate the final prediction. This batch processing approach renders the LLM agnostic to the order of ICL examples. Through extensive experiments and analysis, we demonstrate that Batch-ICL consistently outperforms most permutations of ICL examples. In some cases, it even exceeds the performance of the best order for standard ICL, all while reducing the computational resources required. Furthermore, we develop a novel variant of Batch-ICL featuring multiple "epochs" of meta-optimization. This variant implicitly explores permutations of ICL examples, further enhancing ICL performance.