mlsanityData to 7 Oct 2026

LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation

arXiv 2410.13846 ↗First posted 17 Oct 2024cs.CL cs.AI cs.LGcited by 11

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Scaling language models to handle longer contexts introduces substantial memory challenges due to the growing cost of key-value (KV) caches. Motivated by the efficiency gains of hybrid models and the broad availability of pretrained large transformer backbones, we explore transitioning transformer models into hybrid architectures for a more efficient generation. In this work, we propose LightTransfer, a lightweight method that transforms models such as LLaMA into hybrid variants. Our approach identifies lazy layers -- those focusing on recent or initial tokens -- and replaces their full attention with streaming attention. This transformation can be performed without any training for long-context understanding tasks or with minimal fine-tuning for o1-like long reasoning generation tasks that require stronger reasoning capabilities. Experiments across diverse benchmarks and models (e.g., LLaMA, Mistral, QwQ-STILL) demonstrate that, even when half of the layers are identified as lazy, LightTransfer achieves up to 2.17× throughput improvement with minimal performance loss (<1.5% on LongBench) and achieves 53.3% on math benchmark AIME24 of advanced o1-like long reasoning model QwQ-STILL.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 52 of its 63 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

52 of 63 references.

References · 52

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2023Mistral 7Bcited by 3,065
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
38 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
  11. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  12. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  13. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  14. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  15. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
  16. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  17. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  18. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  19. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 122
  20. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  21. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  22. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  23. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  24. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
  25. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  26. 2021Finetuning Pretrained Transformers into RNNscited by 94
  27. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Taskscited by 94
  28. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
  29. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  30. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  31. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  32. 2024Linearizing Large Language Modelscited by 51
  33. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 48
  34. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  35. 2024Investigating Layer Importance in Large Language Modelscited by 35
  36. 2024Effectively Compress KV Heads for LLMcited by 30
  37. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budgetcited by 21
  38. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressioncited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  2. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  3. 2025The Pitfalls of KV Cache Compressioncited by 15
  4. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  5. 2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4
  6. 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 3
  7. 2025SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processingcited by 3
  8. 2026Memory for Large Language Modelscited by 2
  9. 2026Prefix Sliding for efficient test-time scalingcited by 2
  10. 2026NLL-Guided Full-Attention Layer Selection for Training-Free Sliding-Window Adaptationcited by 1
  11. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LightThinker++: From Reasoning Compression to Memory Management
  2. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
  3. 2026SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers
  4. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading
  5. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
  6. 2025LightThinker: Thinking Step-by-Step Compression
  7. 2026Efficient Reasoning on the Edge
  8. 2025Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers