# Dual Debiasing for Noisy In-Context Learning for Text Generation

arXiv: https://arxiv.org/abs/2506.00418 · HTML: https://mlsanity.com/paper/2506.00418/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 31 May 2025.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Detecting Language Model Attacks with Perplexity](https://mlsanity.com/paper/2308.14132/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [Compositional Exemplars for In-context Learning](https://mlsanity.com/paper/2302.05698/) (2023)
- [Unified Demonstration Retriever for In-Context Learning](https://mlsanity.com/paper/2305.04320/) (2023)
- [Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations](https://mlsanity.com/paper/2205.12685/) (2022)
- [Mitigating Label Biases for In-context Learning](https://mlsanity.com/paper/2305.19148/) (2023)
- [Types of Out-of-Distribution Texts and How to Detect Them](https://mlsanity.com/paper/2109.06827/) (2021)
- [OpenICL: An Open-Source Framework for In-context Learning](https://mlsanity.com/paper/2303.02913/) (2023)
- [Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2405.02750/) (2024)
- [Benchmarking GPT-4 against Human Translators: A Comprehensive Evaluation Across Languages, Domains, and Expertise Levels](https://mlsanity.com/paper/2411.13775/) (2024)

## Built on by (most cited first, 2 of 2)

- [Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions](https://mlsanity.com/paper/2508.13214/) (2025)
- [Retrieval-Guided Fine-Tuning as Noisy Estimation: Risk bounds and Architectural Analysis](https://mlsanity.com/paper/2609.14485/) (2026)

## Abstract (arXiv)

In context learning (ICL) relies heavily on high quality demonstrations drawn from large annotated corpora. Existing approaches detect noisy annotations by ranking local perplexities, presuming that noisy samples yield higher perplexities than their clean counterparts. However, this assumption breaks down when the noise ratio is high and many demonstrations are flawed. We reexamine the perplexity based paradigm for text generation under noisy annotations, highlighting two sources of bias in perplexity: the annotation itself and the domain specific knowledge inherent in large language models (LLMs). To overcome these biases, we introduce a dual debiasing framework that uses synthesized neighbors to explicitly correct perplexity estimates, yielding a robust Sample Cleanliness Score. This metric uncovers absolute sample cleanliness regardless of the overall corpus noise level. Extensive experiments demonstrate our method's superior noise detection capabilities and show that its final ICL performance is comparable to that of a fully clean demonstration corpus. Moreover, our approach remains robust even when noise ratios are extremely high.