# The Sirens' Song: When Proximal Background Context Overshadows Distant Evidence

arXiv: https://arxiv.org/abs/2609.26718 · HTML: https://mlsanity.com/paper/2609.26718/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 26 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention](https://mlsanity.com/paper/2407.02490/) (2024)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [XAttention: Block Sparse Attention with Antidiagonal Scoring](https://mlsanity.com/paper/2503.16428/) (2025)
- [FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inference](https://mlsanity.com/paper/2502.20766/) (2025)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training](https://mlsanity.com/paper/2311.09198/) (2023)
- [Structured Packing in LLM Training Improves Long Context Utilization](https://mlsanity.com/paper/2312.17296/) (2023)
- [Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference](https://mlsanity.com/paper/2512.16391/) (2025)
- [ProxyAttn: Guided Sparse Attention via Representative Heads](https://mlsanity.com/paper/2509.24745/) (2025)
- [Sparser Block-Sparse Attention via Token Permutation](https://mlsanity.com/paper/2510.21270/) (2025)
- [Stem: Rethinking Causal Information Flow in Sparse Attention](https://mlsanity.com/paper/2603.06274/) (2026)
- [S2O: Early Stopping for Sparse Attention via Online Permutation](https://mlsanity.com/paper/2602.22575/) (2026)

## Abstract (arXiv)

Long-context LLMs focus on retrieving distant evidence from extensive context, yet existing work has largely focused on overcoming distance alone. In this work, we identify the Proximity Trap, insufficient attention to distant evidence often arises less from distance itself than from cumulative competition with abundant, task-irrelevant proximal background. To address the Proximity Trap, we introduce LYRA (Long-context heavY-tailed Relevance Alignment), a t-distributed directional matching mechanism that reshapes the context retrieval distribution, directing more attention mass toward task-relevant evidence, while preserving the relative positional information encoded. Extensive experiments on LongBench-v2, RULER, and LongBench demonstrate consistent improvements across context lengths and task categories. We further introduce ProxBench, a multi-level fine-grained benchmark for evaluating distant evidence utilization under increasing proximal background interference. Project page: https://xiaoyuyoung.github.io/LYRA/