mlsanityData to 7 Oct 2026

NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning

arXiv 2402.05515 ↗First posted 8 Feb 2024cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on two models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with more faithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

22 of 43 references.

References · 22

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  4. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  5. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  6. 2021MetaICL: Learning to Learn In Contextcited by 450
  7. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  8. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  9. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  10. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 209
  11. 2018Hate Speech Dataset from a White Supremacy Forumcited by 196
  12. 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
10 more
  1. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  2. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  3. 2023Mitigating Label Biases for In-context Learningcited by 88
  4. 2023Symbol tuning improves in-context learning in language modelscited by 72
  5. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  6. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  7. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  8. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  9. 2022Robust Models are less Over-Confidentcited by 25
  10. 2023Transformer Language Models Handle Word Frequency in Prediction Headcited by 8

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  2. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  3. 2025StaICC: Standardized Evaluation for Classification Task in In-context Learningcited by 2
  4. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 2