mlsanityData to 7 Oct 2026

Long Context Pre-Training with Lighthouse Attention

arXiv 2605.06554 ↗First posted 7 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Training causal transformers at extreme sequence lengths is bottlenecked by the quadratic time and memory of scaled dot-product attention (SDPA). In this work, we propose Lighthouse Attention, a training-only symmetrical selection-based hierarchical attention algorithm that wraps around ordinary SDPA and can be easily removed towards the end of the training. Our hierarchical selection is also gradient-free, which exempts us from dealing with a complicated and potentially inefficient backward pass kernel. Our contribution is three-fold: (i) A subquadratic hierarchical pre- and post-processing step that does adaptive compression and decompression of the sequence. (ii) A symmetrical compression strategy that pools queries, keys and values at the same time, while preserving left-to-right causality, which greatly improves parallelism. (iii) A two stage training approach which we pre-train for the majority of the time with Lighthouse Attention and recover a full attention model at the end with a short training. We run preliminary small scale LLM pre-training experiments that show the effectiveness of our method compared to full attention training with all other settings matched, where we achieve a faster total training time and lower final loss after the recovery phase. Full code is available at: https://github.com/ighoshsubho/lighthouse-attention

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

35 of 40 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  11. 2020Rethinking Attention with Performerscited by 1,584
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
23 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  6. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  13. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  14. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  15. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  16. 2024Transformers are Multi-State RNNscited by 120
  17. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  18. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  19. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  20. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  21. 2025Log-Linear Attentioncited by 39
  22. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  23. 2026HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attentioncited by 10

Built on by

Not cited yet.