mlsanityData to 7 Oct 2026

NICE: To Optimize In-Context Examples or Not?

arXiv 2402.06733 ↗First posted 9 Feb 2024cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recent work shows that in-context learning and optimization of in-context examples (ICE) can significantly improve the accuracy of large language models (LLMs) on a wide range of tasks, leading to an apparent consensus that ICE optimization is crucial for better performance. However, most of these studies assume a fixed or no instruction provided in the prompt. We challenge this consensus by investigating the necessity of optimizing ICE when task-specific instructions are provided and find that there are many tasks for which it yields diminishing returns. In particular, using a diverse set of tasks and a systematically created instruction set with gradually added details, we find that as the prompt instruction becomes more detailed, the returns on ICE optimization diminish. To characterize this behavior, we introduce a task-specific metric called Normalized Invariability to Choice of Examples (NICE) that quantifies the learnability of tasks from a given instruction, and provides a heuristic to help decide whether to optimize instructions or ICE for a new task. Given a task, the proposed metric can reliably predict the utility of optimizing ICE compared to using random ICE. Our code is available at https://github.com/microsoft/nice-icl.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 9 Feb 2024.

Builds on

32 of 45 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
20 more
  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  2. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  3. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  4. 2023Larger language models do in-context learning differentlycited by 386
  5. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  6. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  7. 2022Active Example Selection for In-Context Learningcited by 195
  8. 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 156
  9. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  10. 2023Compositional Exemplars for In-context Learningcited by 143
  11. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  12. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
  13. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  14. 2018NL2Bash: A Corpus and Semantic Parser for Natural Language Interface to the Linux Operating Systemcited by 97
  15. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  16. 2023In-context Example Selection with Influencescited by 60
  17. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  18. 2021Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learningcited by 35
  19. 2023Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Modelscited by 21
  20. 2023How Robust are LLMs to In-Context Majority Label Bias?cited by 13

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 6.

9 papers.

  1. 2025LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacracited by 20
  2. 2024Task Facet Learning: A Structured Approach to Prompt Optimizationcited by 17
  3. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  4. 2025Investigating the Zone of Proximal Development of Language Models for In-Context Learningcited by 6
  5. 2025Number Representations in LLMs: A Computational Parallel to Human Perceptioncited by 4
  6. 2025Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Trainingcited by 3
  7. 2024Large Language Models Might Not Care What You Are Saying: Prompt Format Beats Descriptionscited by 2
  8. 202460 Data Points are Sufficient to Fine-Tune LLMs for Question-Answeringcited by 2
  9. 2025ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Executioncited by 1