# Concise and Precise Context Compression for Tool-Using Language Models

arXiv: https://arxiv.org/abs/2407.02043 · HTML: https://mlsanity.com/paper/2407.02043/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 13 of its 28 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 13 of 13)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)

## Built on by (most cited first, 12 of 12)

- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management](https://mlsanity.com/paper/2510.06727/) (2025)
- [SkillReducer: Optimizing LLM Agent Skills for Token Efficiency](https://mlsanity.com/paper/2603.29919/) (2026)
- [What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices](https://mlsanity.com/paper/2409.01893/) (2024)
- [AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees](https://mlsanity.com/paper/2512.04550/) (2025)
- [Solving Context Window Overflow in AI Agents](https://mlsanity.com/paper/2511.22729/) (2025)
- [NaviAgent: Graph-Driven Bilevel Planning for Scalable Tool Orchestration](https://mlsanity.com/paper/2506.19500/) (2025)
- [From Context to EDUs: Faithful and Structured Context Compression via Elementary Discourse Unit Decomposition](https://mlsanity.com/paper/2512.14244/) (2025)
- [ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL](https://mlsanity.com/paper/2606.31650/) (2026)
- [Control Under Compression: Reliability Frontiers for Tool-Using Agents](https://mlsanity.com/paper/2608.01056/) (2026)
- [ParaTool: Shifting Tool Representations from Context to Parameters](https://mlsanity.com/paper/2605.29561/) (2026)
- [MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression](https://mlsanity.com/paper/2609.27380/) (2026)

## Abstract (arXiv)

Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process. Given the progress in general-purpose compression, soft context compression is a suitable approach to alleviate the problem. However, when compressing tool documentation, existing methods suffer from the weaknesses of key information loss (specifically, tool/parameter name errors) and difficulty in adjusting the length of compressed sequences based on documentation lengths. To address these problems, we propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. 1) Selective compression strategy mitigates key information loss by deliberately retaining key information as raw text tokens. 2) Block compression strategy involves dividing tool documentation into short chunks and then employing a fixed-length compression model to achieve variable-length compression. This strategy facilitates the flexible adjustment of the compression ratio. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.