# TriAttention: Efficient Long Reasoning with Trigonometric KV Compression

arXiv: https://arxiv.org/abs/2604.04921 · HTML: https://mlsanity.com/paper/2604.04921/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 40 of its 45 references that are in the map.
- First posted on arXiv 6 Apr 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Efficient LLM inference and serving / [Efficient reasoning and agent inference](https://mlsanity.com/inference/efficient-reasoning-and-agent-inference/)

## Builds on (most cited first, 30 of 40)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs](https://mlsanity.com/paper/2310.01801/) (2023)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling](https://mlsanity.com/paper/2406.02069/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference](https://mlsanity.com/paper/2407.11550/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [When Attention Sink Emerges in Language Models: An Empirical View](https://mlsanity.com/paper/2410.10781/) (2024)
- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Transformers are Multi-State RNNs](https://mlsanity.com/paper/2401.06104/) (2024)
- [Round and Round We Go! What makes Rotary Positional Encodings useful?](https://mlsanity.com/paper/2410.06205/) (2024)

## Built on by (most cited first, 21 of 21)

- [LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation](https://mlsanity.com/paper/2605.18739/) (2026)
- [Addressable Memory for Video World Models](https://mlsanity.com/paper/2608.07408/) (2026)
- [Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation](https://mlsanity.com/paper/2605.16003/) (2026)
- [MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression](https://mlsanity.com/paper/2604.17695/) (2026)
- [Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning](https://mlsanity.com/paper/2605.09490/) (2026)
- [AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees](https://mlsanity.com/paper/2605.19260/) (2026)
- [Adaptive Mass-Segmented KV Compression for Long-Context Reasoning](https://mlsanity.com/paper/2605.23200/) (2026)
- [RoPE-Aware Bit Allocation for KV-Cache Quantization](https://mlsanity.com/paper/2606.24033/) (2026)
- [LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding](https://mlsanity.com/paper/2607.24555/) (2026)
- [ATFlash: Per-RoPE-Wavelength Attention Windows for Compute/Memory-Efficient LLM Inference](https://mlsanity.com/paper/2608.02947/) (2026)
- [Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning](https://mlsanity.com/paper/2609.03430/) (2026)
- [CASK: Core-Aware Selective KV Compression for Reasoning Traces](https://mlsanity.com/paper/2604.10900/) (2026)
- [HeadQ: Model-Visible Distortion and Score-Space Correction for KV-Cache Quantization](https://mlsanity.com/paper/2605.03562/) (2026)
- [Minimal-Intervention KV Retention via Set-Conditioned Diversity](https://mlsanity.com/paper/2605.14292/) (2026)
- [Precomputing the Future-Offset Average in TriAttention](https://mlsanity.com/paper/2607.13051/) (2026)
- [CommitKV: Lifecycle-Aware KV Cache Compression via Commit Transitions for Multi-Turn Agents](https://mlsanity.com/paper/2608.07855/) (2026)
- [AgentKV: Phase-Aware KV Eviction for Agentic LLMs](https://mlsanity.com/paper/2609.14872/) (2026)
- [Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning](https://mlsanity.com/paper/2610.05685/) (2026)
- [WAM-Cache: Staleness-Bounded KV Reuse for Efficient World Action Models](https://mlsanity.com/paper/2610.11401/) (2026)
- [GateSight: Recurrent Memory Anticipates Attention Memory for Query-Agnostic KV Eviction in Hybrid LLMs](https://mlsanity.com/paper/2610.14754/) (2026)
- [Joint Subspace Sparse Attention for Efficient Long-Context Decoding](https://mlsanity.com/paper/2610.19026/) (2026)

## Abstract (arXiv)

Extended reasoning in large language models (LLMs) creates severe KV cache memory bottlenecks. Leading KV cache compression methods estimate KV importance using attention scores from recent post-RoPE queries. However, queries rotate with position during RoPE, making representative queries very few, leading to poor top-key selection and unstable reasoning. To avoid this issue, we turn to the pre-RoPE space, where we observe that Q and K vectors are highly concentrated around fixed non-zero centers and remain stable across positions -- Q/K concentration. We show that this concentration causes queries to preferentially attend to keys at specific distances (e.g., nearest keys), with the centers determining which distances are preferred via a trigonometric series. Based on this, we propose TriAttention to estimate key importance by leveraging these centers. Via the trigonometric series, we use the distance preference characterized by these centers to score keys according to their positions, and also leverage Q/K norms as an additional signal for importance estimation. On AIME25 with 32K-token generation, TriAttention matches Full Attention reasoning accuracy while achieving 2.5x higher throughput or 10.7x KV memory reduction, whereas leading baselines achieve only about half the accuracy at the same efficiency. TriAttention enables OpenClaw deployment on a single consumer GPU, where long context would otherwise cause out-of-memory with Full Attention.