mlsanityData to 7 Oct 2026

Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning

arXiv 2401.06469 ↗First posted 12 Jan 2024cs.LG cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to the development of Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for ICL. Differing from the standard N-shot learning approach, Batch-ICL employs N separate 1-shot forward computations and aggregates the resulting meta-gradients. These aggregated meta-gradients are then applied to the forward computation of a zero-shot query to generate the final prediction. This batch processing approach renders the LLM agnostic to the order of ICL examples. Through extensive experiments and analysis, we demonstrate that Batch-ICL consistently outperforms most permutations of ICL examples. In some cases, it even exceeds the performance of the best order for standard ICL, all while reducing the computational resources required. Furthermore, we develop a novel variant of Batch-ICL featuring multiple "epochs" of meta-optimization. This variant implicitly explores permutations of ICL examples, further enhancing ICL performance.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 12 Jan 2024.

Builds on

21 of 31 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  10. 2022In-context Learning and Induction Headscited by 930
  11. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  12. 2022Transformers learn in-context by gradient descentcited by 714
9 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2021True Few-Shot Learning with Language Modelscited by 343
  3. 2023In-Context Learning Creates Task Vectorscited by 302
  4. 2023Function Vectors in Large Language Modelscited by 292
  5. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  6. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  7. 2022Parallel Context Windows for Large Language Modelscited by 81
  8. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 45
  9. 2023Towards Understanding How Transformers Learn In-context Through a Representation Learning Lenscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 7; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  2. 2024From Skepticism to Acceptance: Simulating the Attitude Dynamics Toward Fake Newscited by 51
  3. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reasoncited by 37
  4. 2024Finding Visual Task Vectorscited by 24
  5. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Modelscited by 19
  6. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  7. 2025More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectivescited by 12
  8. 2025Rethinking Invariance in In-context Learningcited by 8
  9. 2025OptiSeq: Ordering Examples On-The-Fly for In-Context Learningcited by 6
  10. 2024Language Models "Grok" to Copycited by 5
  11. 2025ReFeed: Multi-dimensional Summarization Refinement with Reflective Reasoning on Feedbackcited by 4
  12. 2025Adaptive Task Vectors for Large Language Modelscited by 4
8 more
  1. 2025Hearing the Order: Investigating Position Bias in Large Audio-Language Modelscited by 4
  2. 2024Disentangling Latent Shifts of In-Context Learning with Weak Supervisioncited by 3
  3. 2024MILE: A Mutation Testing Framework of In-Context Learning Systemscited by 2
  4. 2026PLR: Plackett-Luce for Reordering In-Context Learning Examplescited by 1
  5. 2026Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selectioncited by 1
  6. 2024Distributed In-Context Learning under Non-IID Among Clientscited by 0
  7. 2026Self-Improving In-Context Learningcited by 0
  8. 2026Efficient Task Adaptation in Large Language Models: A Survey of Weight-Based, Prompt-Based, and Embedding-Based Adaptationscited by 0