mlsanityData to 7 Oct 2026

TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

arXiv 2604.04921 ↗First posted 6 Apr 2026cs.CL cs.CVcited by 18

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable reasoning. To avoid this issue, we turn to the pre-RoPE space, where we observe that Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions -- Q/K concentration. We show that this concentration causes queries to preferentially attend to keys at specific distances (e.g., nearest keys), with the centers determining which distances are preferred via a trigonometric series. Based on this, we propose TriAttention to estimate key importance by leveraging these centers. Via the trigonometric series, we use the distance preference characterized by these centers to score keys according to their positions, and also leverage Q/K norms as an additional signal for importance estimation. On AIME25 with 32K-token generation, TriAttention matches Full Attention reasoning accuracy while achieving 2.5x higher throughput or 10.7x KV memory reduction, whereas leading baselines achieve only about half the accuracy at the same efficiency. TriAttention enables OpenClaw deployment on a single consumer GPU, where long context would otherwise cause out-of-memory with Full Attention.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 40 of its 45 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

40 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug
  2. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning19 Jun
  3. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Models30 May
  4. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning16 Feb
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec

Other references · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2025Gemma 3 Technical Reportcited by 1,686
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
23 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  7. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  8. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  9. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  10. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  11. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  12. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  13. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  14. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  15. 2024Transformers are Multi-State RNNscited by 120
  16. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  17. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  18. 2025Scaling RL to Long Videoscited by 83
  19. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  20. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  21. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  22. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  23. 2025EliteKV: Scalable KV Cache Compression via RoPE Frequency Selection and Joint Low-Rank Projectioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generationcited by 25
  2. 2026Addressable Memory for Video World Modelscited by 11
  3. 2026MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compressioncited by 1
  4. 2026Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoningcited by 1
  5. 2026AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtreescited by 1
  6. 2026Adaptive Mass-Segmented KV Compression for Long-Context Reasoningcited by 1
  7. 2026RoPE-Aware Bit Allocation for KV-Cache Quantizationcited by 1
  8. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  9. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoningcited by 1
  10. 2026CASK: Core-Aware Selective KV Compression for Reasoning Tracescited by 0
  11. 2026HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantizationcited by 0
  12. 2026Minimal-Intervention KV Retention via Set-Conditioned Diversitycited by 0
6 more
  1. 2026Precomputing the Future-Offset Average in TriAttentioncited by 0
  2. 2026ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inferencecited by 0
  3. 2026CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agentscited by 0
  4. 2026AgentKV: Phase-Aware KV Eviction for Agentic LLMscited by 0
  5. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0
  6. 2026WAM-Cache: Staleness-Bounded KV Reuse for Efficient World Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LongFlow: Efficient KV Cache Compression for Reasoning Models
  2. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  3. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
  4. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning
  5. 2026Information-Aware KV Cache Compression for Long Reasoning
  6. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression
  7. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
  8. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving