Data to 13 Oct 2026

You Only Cache Once: Decoder-Decoder Architectures for Language Models

arXiv 2405.05254 ↗First posted 8 May 2024cs.CLcited by 139

Not on the map. Why?

Cited by 62 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a decoder-decoder architecture, YOCO, for large language models, which only caches key-value pairs once. It consists of two components, i.e., a cross-decoder stacked upon a self-decoder. The self-decoder efficiently encodes global key-value (KV) caches that are reused by the cross-decoder via cross-attention. The overall model behaves like a decoder-only Transformer, although YOCO only caches once. The design substantially reduces GPU memory demands, yet retains global attention capability. Additionally, the computation flow enables prefilling to early exit without changing the final output, thereby significantly speeding up the prefill stage. Experimental results demonstrate that YOCO achieves favorable performance compared to Transformer in various settings of scaling up model size and number of training tokens. We also extend YOCO to 1M context length with near-perfect needle retrieval accuracy. The profiling results show that YOCO improves inference memory, prefill latency, and throughput by orders of magnitude across context lengths and model sizes. Code is available at https://aka.ms/YOCO.

Key facts. Cited by 139 papers in the map (data through 13 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

43 of 55 references.

References · 43

2017Attention Is All You Needcited by 60,106
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
2020Language Models are Few-Shot Learnerscited by 32,320
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
2020Scaling Laws for Neural Language Modelscited by 6,494
2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,658
2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,628
2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
2020GLU Variants Improve Transformercited by 1,759

31 more

2019Generating Long Sequences with Sparse Transformerscited by 1,538
2018Group Normalizationcited by 1,518
2019Root Mean Square Layer Normalizationcited by 1,449
2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,115
2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 813
2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 708
2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 609
2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 603
2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 589
2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 572
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
2022Efficiently Scaling Transformer Inferencecited by 532
2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 500
2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 452
2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 394
2024xLSTM: Extended Long Short-Term Memorycited by 322
2023Effective Long-Context Scaling of Foundation Modelscited by 302
2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 202
2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 192
2023Augmenting Language Models with Long-Term Memorycited by 188
2024Data Engineering for Scaling Language Models to 128K Contextcited by 185
2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 176
2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 46
2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 36; 2025: 61; 2026: 42.

139 papers.

Sort:

By area:

2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 501
2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 378
2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 378
2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 178
2024How to Train Long-Context Language Models (Effectively)cited by 144
2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 126
2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 125
2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 74
2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 74
2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 58
2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferencescited by 58

38 more

2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 57
2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
2024Multimodal Latent Language Modeling with Next-Token Diffusioncited by 47
2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
2024LinFusion: 1 GPU, 1 Minute, 16K Imagecited by 42
2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
2024Memory3: Language Modeling with Explicit Memorycited by 40
2024KVSharer: Efficient Inference via Layer-Wise Dissimilar KV Cache Sharingcited by 39
2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 36
2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
2024MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projectioncited by 29
2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 29
2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 28
2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 28
2024Q-Sparse: All Large Language Models can be Fully Sparsely-Activatedcited by 25
2024Star Attention: Efficient LLM Inference over Long Sequencescited by 25
2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 24
2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 22
2024MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decodingcited by 21
2025Autoregressive Image Generation with Randomized Parallel Decodingcited by 19
2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
2024A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inferencecited by 16
2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16
2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
2025Lookahead Q-Cache: Achieving More Consistent KV Cache Eviction via Pseudo Querycited by 14
2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
2025Gating is Weighting: Understanding Gated Linear Attention through In-context Learningcited by 12
2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 10

50 most cited of 139 All 139 (JSON)

Similar, not cited

Close in wording, no citation either way