mlsanityData to 14 Oct 2026

SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator

arXiv 2412.12094 ↗First posted 16 Dec 2024cs.CL cs.AI cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have exhibited exceptional performance across a spectrum of natural language processing tasks. However, their substantial sizes pose considerable challenges, particularly in computational demands and inference speed, due to their quadratic complexity. In this work, we have identified a key pattern: certain seemingly meaningless separator tokens (i.e., punctuations) contribute disproportionately to attention scores compared to semantically meaningful tokens. This observation suggests that information of the segments between these separator tokens can be effectively condensed into the separator tokens themselves without significant information loss. Guided by this insight, we introduce SepLLM, a plug-and-play framework that accelerates inference by compressing these segments and eliminating redundant tokens. Additionally, we implement efficient kernels for training acceleration. Experimental results across training-free, training-from-scratch, and post-training settings demonstrate SepLLM's effectiveness. Notably, using the Llama-3-8B backbone, SepLLM achieves over 50% reduction in KV cache on the GSM8K-CoT benchmark while maintaining comparable performance. Furthermore, in streaming settings, SepLLM effectively processes sequences of up to 4 million tokens or more while maintaining consistent language modeling capabilities.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 35 of its 46 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

35 of 46 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
23 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  12. 2023The Falcon Series of Open Language Modelscited by 535
  13. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  14. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  15. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  16. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  17. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  18. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  19. 2020Transformers for Modeling Physical Systemscited by 106
  20. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
  21. 2021SparseBERT: Rethinking the Importance Analysis in Self-attentioncited by 46
  22. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  23. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 21; 2026: 19.

41 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  2. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  3. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 109
  4. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  5. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Modelscited by 54
  6. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  7. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  8. 2025Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routingcited by 14
  9. 2025Thinking Inside the Mask: In-Place Prompting in Diffusion LLMscited by 13
  10. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 13
  11. 2026IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruningcited by 12
  12. 2025SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Cachingcited by 10
29 more
  1. 2026Dual Latent Memory for Visual Multi-agent Systemcited by 10
  2. 2025GEM: Empowering LLM for both Embedding Generation and Language Understandingcited by 8
  3. 2025OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inferencecited by 8
  4. 2025A3: Attention-Aware Accurate KV Cache Fusion for Fast Large Language Model Servingcited by 8
  5. 2025ReMatch: Boosting Representation through Matching for Multimodal Retrievalcited by 8
  6. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  7. 2025Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMscited by 7
  8. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  9. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapsecited by 5
  10. 2026SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Mergingcited by 5
  11. 2025Memory Retrieval and Consolidation in Large Language Models through Function Tokenscited by 4
  12. 2026LightThinker++: From Reasoning Compression to Memory Managementcited by 4
  13. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  14. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  15. 2025Self-Adjust Softmaxcited by 3
  16. 2025Sentence-Anchored Gist Compression for Long-Context LLMscited by 3
  17. 2026SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMscited by 2
  18. 2025Neural Attention Searchcited by 1
  19. 2025CaliDrop: KV Cache Compression with Calibrationcited by 1
  20. 2026Efficient Sequential Recommendation for Long Term User Interest Via Personalizationcited by 1
  21. 2026Dynamic Thinking-Token Selection for Efficient Reasoning in Large Reasoning Modelscited by 1
  22. 2026A State-Transition Framework for Efficient LLM Reasoningcited by 1
  23. 2025ChunkLLM: A Lightweight Pluggable Framework for Accelerating LLMs Inferencecited by 0
  24. 2026ZoomR: Memory Efficient Reasoning through Multi-Granularity Key Value Retrievalcited by 0
  25. 2026HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Stepscited by 0
  26. 2026LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoningcited by 0
  27. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0
  28. 2026Rethinking Contextualization by Reinterpreting Attention Head Channelscited by 0
  29. 2026OmniTide: Co-Designing Algorithms and Systems for Efficient On-Device Omni-LLM Streamingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
  2. 2024A Survey on Efficient Inference for Large Language Models
  3. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormer
  4. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
  5. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  6. 2025Pruning Laws for Large Language Models
  7. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
  8. 2026POP: Prefill-Only Pruning for Efficient Large Model Inference