mlsanityData to 7 Oct 2026

ICLEval: Evaluating In-Context Learning Ability of Large Language Models

arXiv 2406.14955 ↗First posted 21 Jun 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

In-Context Learning (ICL) is a critical capability of Large Language Models (LLMs) as it empowers them to comprehend and reason across interconnected inputs. Evaluating the ICL ability of LLMs can enhance their utilization and deepen our understanding of how this ability is acquired at the training stage. However, existing evaluation frameworks primarily focus on language abilities and knowledge, often overlooking the assessment of ICL ability. In this work, we introduce the ICLEval benchmark to evaluate the ICL abilities of LLMs, which encompasses two key sub-abilities: exact copying and rule learning. Through the ICLEval benchmark, we demonstrate that ICL ability is universally present in different LLMs, and model size is not the sole determinant of ICL efficacy. Surprisingly, we observe that ICL abilities, particularly copying, develop early in the pretraining process and stabilize afterward. Our source codes and benchmark are released at https://github.com/yiye3/ICLEval.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 21 Jun 2024.

Builds on

46 of 61 references.

References · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
34 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  13. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  14. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  15. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  16. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  17. 2018CoQA: A Conversational Question Answering Challengecited by 674
  18. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  19. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  20. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  21. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  22. 2021MetaICL: Learning to Learn In Contextcited by 450
  23. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  24. 2023Faithful Chain-of-Thought Reasoningcited by 285
  25. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
  26. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  27. 2021Meta-learning via Language Model In-context Tuningcited by 163
  28. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  29. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  30. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  31. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  32. 2023OpenICL: An Open-Source Framework for In-context Learningcited by 51
  33. 2022Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibrationcited by 24
  34. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 4.

12 papers.

  1. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 23
  2. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  3. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  4. 2025Advancing Multimodal In-Context Learning in Large Vision-Language Models with Task-aware Demonstrationscited by 5
  5. 2025A Single Character can Make or Break Your LLM Evalscited by 4
  6. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 4
  7. 2024MILE: A Mutation Testing Framework of In-Context Learning Systemscited by 2
  8. 2025SynSpill: Improved Industrial Spill Detection With Synthetic Datacited by 1
  9. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  10. 2026When Demonstrations Fail: Diagnosing the Limits of In-Context Learning in Large Audio-Language Models with Progressive Cue Removalcited by 0
  11. 2026Self-Improving In-Context Learningcited by 0
  12. 2026When Context Misleads: In-context Learning with Jurisdiction in Large Language Modelscited by 0