# When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaboration

arXiv: https://arxiv.org/abs/2604.13349 · HTML: https://mlsanity.com/paper/2604.13349/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time](https://mlsanity.com/paper/2305.17118/) (2023)
- [RazorAttention: Efficient KV Cache Compression Through Retrieval Heads](https://mlsanity.com/paper/2407.15891/) (2024)
- [Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning](https://mlsanity.com/paper/2410.19258/) (2024)
- [DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs](https://mlsanity.com/paper/2412.14838/) (2024)
- [Augmenting Multi-Agent Communication with State Delta Trajectory](https://mlsanity.com/paper/2506.19209/) (2025)
- [Interpreting and Exploiting Functional Specialization in Multi-Head Attention under Multi-task Learning](https://mlsanity.com/paper/2310.10318/) (2023)
- [Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inference](https://mlsanity.com/paper/2601.07667/) (2026)

## Built on by (most cited first, 5 of 5)

- [Do Latent Channels Actually Communicate? A Causal Audit of Latent Multi-Agent LLM](https://mlsanity.com/paper/2607.26773/) (2026)
- [When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs](https://mlsanity.com/paper/2608.04893/) (2026)
- [Semantic Register Compression in Multi-Agent LLM Cascades](https://mlsanity.com/paper/2607.14119/) (2026)
- [Receiver-Conditioned Latent Communication gives 94% CacheBack](https://mlsanity.com/paper/2609.32046/) (2026)
- [Draft-KV: Learning Useful Latent Communication Between Language Models](https://mlsanity.com/paper/2609.34754/) (2026)

## Abstract (arXiv)

Multi-agent LLM systems are moving beyond discrete-token messages toward richer relays that preserve internal state. Recent work such as LatentMAS transmits full key-value (KV) caches between agents but pays a high memory and communication cost. We adapt KV-cache eviction to this setting and introduce Orthogonal BackFill (OBF), which injects a low-rank residual from the discarded KV states back into the retained ones, orthogonal to what is already kept. With only 9.9%-20.2% of the prompt KV retained, compressed relay cuts bandwidth by 4.7× and GPU memory by 8% at under 5% wall-clock overhead, and stays close to full relay in accuracy across nine benchmarks, ahead of it on several. OBF matches or improves over headwise eviction on all nine, and its gain is proportional to the accuracy gap eviction opens against full relay (r=0.78 across three model scales), so it gives back part of what eviction takes. Code is available at https://github.com/markli404/When-Less-Latent-Leads-to-Better-Relay.