# On Problems of Implicit Context Compression for Software Engineering Agents

arXiv: https://arxiv.org/abs/2605.11051 · HTML: https://mlsanity.com/paper/2605.11051/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 34 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 30)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [SWE-smith: Scaling Data for Software Engineering Agents](https://mlsanity.com/paper/2504.21798/) (2025)
- [CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation](https://mlsanity.com/paper/2502.21074/) (2025)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention](https://mlsanity.com/paper/2404.07143/) (2024)
- [Text Embeddings Reveal (Almost) As Much As Text](https://mlsanity.com/paper/2310.06816/) (2023)
- [RedPajama: an Open Dataset for Training Large Language Models](https://mlsanity.com/paper/2411.12372/) (2024)
- [ACON: Optimizing Context Compression for Long-horizon LLM Agents](https://mlsanity.com/paper/2510.00615/) (2025)
- [SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs](https://mlsanity.com/paper/2502.12134/) (2025)
- [Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents](https://mlsanity.com/paper/2509.23040/) (2025)
- [RepoQA: Evaluating Long Context Code Understanding](https://mlsanity.com/paper/2406.06025/) (2024)
- [Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings](https://mlsanity.com/paper/2512.12167/) (2025)
- [Natural Is The Best: Model-Agnostic Code Simplification for Pre-trained Large Language Models](https://mlsanity.com/paper/2405.11196/) (2024)
- [Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models](https://mlsanity.com/paper/2409.14364/) (2024)
- [Solving Context Window Overflow in AI Agents](https://mlsanity.com/paper/2511.22729/) (2025)

## Abstract (arXiv)

LLM-based Software Engineering agents face a critical bottleneck: context length limitations cause failures on complex, long-horizon tasks. One promising solution is to encode context as continuous embeddings rather than discrete tokens, enabling denser information storage. We apply the recently proposed In-Context Autoencoder for this purpose. While the method performs well on single-shot common-knowledge and code-understanding tasks, our experiments demonstrate that it fails on multi-step agentic coding tasks. In this paper, we explore this phenomenon and discuss possible factors contributing to this failure.