mlsanityData to 14 Oct 2026

Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing

arXiv 2605.10544 ↗First posted 11 May 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Long-context adaptation is often viewed as window scaling, but this misses a token-level supervision mismatch: in packed training with document masking, each target token's effective context remains short. We introduce EXACT, a supervision-allocation objective that assigns extra weight to long effective-context targets by inverse frequency within the long tail. Across seven Qwen/LLaMA CPT configurations, EXACT improves all 28 trained/extrapolated NoLiMa and RULER comparisons. On Qwen2.5-0.5B, NoLiMa improves by +10.09 (trained) and +5.34 (extrapolated); RULER by +10.69 and +5.55. On LLaMA-3.2-3B, RULER improves by +17.91 and +16.11. Standard QA/reasoning are preserved (+0.24 macro change across six benchmarks). A distance-resolved probe shows gains arise when evidence is thousands of tokens away, while short cases remain unchanged. Results support a supervision-centric thesis: long-context adaptation depends on how strongly training supervises long-context predictions.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

45 of 51 references.

References · 45

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2020Rethinking Attention with Performerscited by 1,601
33 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2016Training Region-based Object Detectors with Online Hard Example Miningcited by 579
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  11. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  12. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  13. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  14. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  15. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  16. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  17. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  18. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  19. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  20. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  21. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
  22. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  23. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  24. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  25. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  26. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  27. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  28. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  29. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  30. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  31. 2024Question Calibration and Multi-Hop Modeling for Temporal Question Answeringcited by 25
  32. 2025Token Weighting for Long-Range Language Modelingcited by 14
  33. 2025StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matchingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corporacited by 1
  2. 2026Scope3Trace: Evidence-Based Identification and Extraction of Scope 3 GHG Emissions from Sustainability Reportscited by 0
  3. 2026Behaviorally Effective LoRA Writes Are Sparse and Structuredcited by 0
  4. 2026Learning Evidence Sufficiency Boundaries for Selective Answering in Grounded Multi-Hop QAcited by 0