mlsanityData to 14 Oct 2026

Refract ICL: Rethinking Example Selection in the Era of Million-Token Models

arXiv 2506.12346 ↗First posted 14 Jun 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The emergence of long-context large language models (LLMs) has enabled the use of hundreds, or even thousands, of demonstrations for in-context learning (ICL) - a previously impractical regime. This paper investigates whether traditional ICL selection strategies, which balance the similarity of ICL examples to the test input (using a text retriever) with diversity within the ICL set, remain effective when utilizing a large number of demonstrations. Our experiments demonstrate that, while longer contexts can accommodate more examples, simply increasing the number of demonstrations does not guarantee improved performance. Smart ICL selection remains crucial, even with thousands of demonstrations. To further enhance ICL in this setting, we introduce Refract ICL, a novel ICL selection algorithm specifically designed to focus LLM attention on challenging examples by strategically repeating them within the context and incorporating zero-shot predictions as error signals. Our results show that Refract ICL significantly improves the performance of extremely long-context models such as Gemini 1.5 Pro, particularly on tasks with a smaller number of output classes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 26 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

8 of 26 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023PaLM 2 Technical Report17 May

Other references · 7

  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  3. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  4. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  5. 2021I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviewscited by 45
  6. 2023Ambiguity-Aware In-Context Learning with Large Language Modelscited by 11
  7. 2023Improving Input-label Mapping with Demonstration Replay for In-context Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models Know What Makes Exemplary Contexts
  2. 2023Dr.ICL: Demonstration-Retrieved In-context Learning
  3. 2023In-Context Learning with Iterative Demonstration Selection
  4. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  5. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
  6. 2024Focused Large Language Models are Stable Many-Shot Learners
  7. 2025On the Relationship Between the Choice of Representation and In-Context Learning
  8. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation