# Lost in the Passage: Passage-level In-context Learning Does Not Necessarily Need a "Passage"

arXiv: https://arxiv.org/abs/2502.10634 · HTML: https://mlsanity.com/paper/2502.10634/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 15 Feb 2025.

## Builds on (most cited first, 17 of 17)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers](https://mlsanity.com/paper/2212.10559/) (2022)
- [Learning to Compress Prompt in Natural Language Formats](https://mlsanity.com/paper/2402.18700/) (2024)
- [Extending Context Window of Large Language Models via Semantic Compression](https://mlsanity.com/paper/2312.09571/) (2023)
- [Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs](https://mlsanity.com/paper/2406.02376/) (2024)
- [Dodo: Dynamic Contextual Compression for Decoder-only LMs](https://mlsanity.com/paper/2310.02409/) (2023)
- [Unsupervised Distractor Generation via Large Language Model Distilling and Counterfactual Contrastive Decoding](https://mlsanity.com/paper/2406.01306/) (2024)

## Built on by (most cited first, 2 of 2)

- [TabSieve: Explicit In-Table Evidence Selection for Tabular Prediction](https://mlsanity.com/paper/2602.11700/) (2026)
- [LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.19614/) (2025)

## Abstract (arXiv)

By simply incorporating demonstrations into the context, in-context learning (ICL) enables large language models (LLMs) to yield awesome performance on many tasks. In this study, we focus on passage-level long-context ICL for generation tasks and find that LLMs cannot learn the intrinsic relationship between the demonstration passage and the generation output. We conduct experiments with different LLMs on two typical generation tasks including single-document question answering and distractor generation, demonstrating that even a completely meaningless demonstration passage with 1/4 length achieves much better performance than the original full passage. Analysis via attention and information flow reveals that LLMs pay little attention to passages compared to other components in the prompt and little information flows from the passage to other parts of the demonstration, which further confirms our finding. Additionally, experiments on context compression indicate that compression approaches proven effective on other long-context tasks are not suitable for passage-level ICL, since simply using shorter meaningless demonstration passages already achieves competitive performance.