# YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference

arXiv: https://arxiv.org/abs/2604.13556 · HTML: https://mlsanity.com/paper/2604.13556/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [You Only Cache Once: Decoder-Decoder Architectures for Language Models](https://mlsanity.com/paper/2405.05254/) (2024)
- [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://mlsanity.com/paper/2405.12981/) (2024)
- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [Hyper-Connections](https://mlsanity.com/paper/2409.19606/) (2024)
- [Attention Residuals](https://mlsanity.com/paper/2603.15031/) (2026)
- [Layer-Condensed KV Cache for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2405.10637/) (2024)
- [DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging](https://mlsanity.com/paper/2402.02622/) (2024)
- [Value Residual Learning](https://mlsanity.com/paper/2410.17897/) (2024)
- [A Systematic Study of Cross-Layer KV Sharing for Efficient LLM Inference](https://mlsanity.com/paper/2410.14442/) (2024)
- [Reconstructing KV Caches with Cross-layer Fusion For Enhanced Transformers](https://mlsanity.com/paper/2512.03870/) (2025)
- [Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads](https://mlsanity.com/paper/2510.16807/) (2025)

## Built on by (most cited first, 2 of 2)

- [CORAL-KV: Compact Cross-Layer Prediction and Residual Coding for Autoregressive Video KV Caches](https://mlsanity.com/paper/2610.12514/) (2026)
- [Orthros: Phase-Aware Heterogeneous Attention for Efficient Transformers](https://mlsanity.com/paper/2610.13735/) (2026)

## Abstract (arXiv)

Cross-layer key-value (KV) compression has been found to be effective in efficient inference of large language models (LLMs). Although they reduce the memory consumption of the KV cache, such methods usually introduce non-negligible performance degradation. In this work, we aim to enhance the performance of YOCO, a cross-layer KV compression method that shares the KVs of the middle layer with the top-half layers. We propose YOCO++, an enhanced YOCO that incorporates a weighted residual connection between the KVs of each bottom-half layer and the bottom layer. Compared to YOCO, YOCO++ increases model capacity while maintaining the same training and inference efficiency. Our experiments show that YOCO++ achieves state-of-the-art performance among the cross-layer KV compression methods at a 50% KV cache compression rate, outperforming the standard Transformer.