mlsanityData to 7 Oct 2026

Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations

arXiv 2503.03062 ↗First posted 4 Mar 2025cs.CL cs.AI cs.IRcited by 7

Not on the map. Why?

Abstract (arXiv)

The high cost of obtaining high-quality annotated data for in-context learning (ICL) has motivated the development of methods that use self-generated annotations in place of ground-truth labels. While these approaches have shown promising results in few-shot settings, they generally do not scale to many-shot scenarios. In this work, we study ICL with self-generated examples using a framework analogous to traditional semi-supervised learning, consisting of annotation generation, demonstration selection, and in-context inference. Within this framework, we propose a simple baseline that outperforms ground-truth ICL in zero-shot, few-shot, and many-shot settings. Notably, we observe a scaling law with this baseline, where optimal performance is achieved with more than 1,000 demonstrations. To fully exploit the many-shot capabilities of semi-supervised ICL, we introduce IterPSD, an iterative annotation approach that integrates iterative refinement and curriculum pseudo-labeling techniques from semi-supervised learning, yielding up to 6.8% additional gains on classification tasks.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

28 of 41 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  8. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  11. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  12. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
16 more
  1. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  2. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  3. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  4. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  5. 2021Curriculum Learning: A Surveycited by 257
  6. 2024Many-Shot In-Context Learningcited by 203
  7. 2019Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detectioncited by 54
  8. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  9. 2023MoT: Memory-of-Thought Enables ChatGPT to Self-Improvecited by 37
  10. 2023JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classificationcited by 15
  11. 2023DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bankcited by 10
  12. 2025TabGen-ICL: Residual-Aware In-Context Example Selection for Tabular Data Generationcited by 10
  13. 2025GLEAN: Active Generalized Category Discovery with Diverse LLM Feedbackcited by 6
  14. 2024Self-Augmented In-Context Learning for Unsupervised Word Translationcited by 4
  15. 2025TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistencycited by 4
  16. 2024PICLe: Pseudo-Annotations for In-Context Learning in Low-Resource Named Entity Detectioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  2. 2025From Web Search towards Agentic Deep Research: Incentivizing Search with Reasoning Agentscited by 49
  3. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  4. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretrainingcited by 3
  5. 2026VIOLA: Towards Video In-Context Learning with Minimal Annotationscited by 2
  6. 2025Chirp Localization via Fine-Tuned Transformer Model: A Proof-of-Concept Studycited by 1
  7. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1