mlsanityData to 7 Oct 2026

TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

arXiv 2605.17170 ↗First posted 16 May 2026cs.LGcited by 3

On the map: LLM agents and tool use → Software engineering coding agents

also in 1 more
Abstract (arXiv)

Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bitwidths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5× KV cache size and achieving 30% higher end-to-end throughput, when running on real GPU systems.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 44 of its 50 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

44 of 50 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware10 Feb

Other references · 43

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2025Qwen3-VL Technical Reportcited by 2,534
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
31 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  6. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  7. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  9. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  11. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  12. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  13. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  14. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  15. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116
  16. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  17. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  18. 2026FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scalingcited by 58
  19. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  20. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  21. 2025LongCodeBench: Evaluating Coding LLMs at 1M Context Windowscited by 37
  22. 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agentscited by 37
  23. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  24. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  25. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  26. 2023Benchmarking LLM powered Chatbots: Methods and Metricscited by 17
  27. 2025LIMINAL: Exploring The Frontiers of LLM Decode Performancecited by 9
  28. 2025Combating the Memory Walls: Optimization Pathways for Long-Context Agentic LLM Inferencecited by 7
  29. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  30. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  31. 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Servingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Anatomy of a Quantized Agent: VRAM Stability and Forecasting in Code-Synthesis Agentic Workloadscited by 0
  2. 2026Parser States Already Know: Structure-Conditioned KV Persistence for Structured Generationcited by 0
  3. 2026Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026PolyKV: A Shared Asymmetrically-Compressed KV Cache Pool for Multi-Agent LLM Inference
  2. 2026StepKV: Step-Aware KV Cache Compression for LLM Agents
  3. 2026AgentKVShift: Efficient KV Cache Reuse for Agentic Memory Systems
  4. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
  5. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inference
  6. 2026IntentKV: Cross-Turn Intent-Aware KV Cache Pruning for Agent Inference
  7. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices
  8. 2026RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse