# TACO-RL: Task Aware Prompt Compression Optimization with Reinforcement Learning

arXiv: https://arxiv.org/abs/2409.13035 · HTML: https://mlsanity.com/paper/2409.13035/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [MeetingBank: A Benchmark Dataset for Meeting Summarization](https://mlsanity.com/paper/2305.17529/) (2023)
- [Learning to Compress Prompt in Natural Language Formats](https://mlsanity.com/paper/2402.18700/) (2024)
- [Discrete Prompt Compression with Reinforcement Learning](https://mlsanity.com/paper/2308.08758/) (2023)
- [Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4](https://mlsanity.com/paper/2401.13810/) (2024)
- [Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning](https://mlsanity.com/paper/2312.08901/) (2023)
- [X-lifecycle Learning for Cloud Incident Management using LLMs](https://mlsanity.com/paper/2404.03662/) (2024)
- [An Empirical Comparison of LM-based Question and Answer Generation Methods](https://mlsanity.com/paper/2305.17002/) (2023)

## Built on by (most cited first, 22 of 22)

- [Prompt Compression for Large Language Models: A Survey](https://mlsanity.com/paper/2410.12388/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [Toward Reliable Context Compression for Long-Horizon Agents: An Empirical Study of Execution Instability](https://mlsanity.com/paper/2608.06503/) (2026)
- [Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention](https://mlsanity.com/paper/2505.15774/) (2025)
- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [Dynamic Compressing Prompts for Efficient Inference of Large Language Models](https://mlsanity.com/paper/2504.11004/) (2025)
- [PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression](https://mlsanity.com/paper/2504.16574/) (2025)
- [Lossless Token Sequence Compression via Meta-Tokens](https://mlsanity.com/paper/2506.00307/) (2025)
- [How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation](https://mlsanity.com/paper/2510.21148/) (2025)
- [What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents](https://mlsanity.com/paper/2607.08032/) (2026)
- [The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts](https://mlsanity.com/paper/2602.15843/) (2026)
- [BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection](https://mlsanity.com/paper/2603.19635/) (2026)
- [Cmprsr: Abstractive Token-Level Question-Agnostic Prompt Compressor](https://mlsanity.com/paper/2511.12281/) (2025)
- [DiffuMask: Diffusion Language Model for Token-level Prompt Pruning](https://mlsanity.com/paper/2604.06627/) (2026)
- [WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference](https://mlsanity.com/paper/2503.17922/) (2025)
- [Seq2Seq2Seq: Lossless Data Compression via Discrete Latent Transformers and Reinforcement Learning](https://mlsanity.com/paper/2602.12146/) (2026)
- [A Unified Model and Document Representation for On-Device Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.14403/) (2026)
- [Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Models](https://mlsanity.com/paper/2604.15153/) (2026)
- [Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compression](https://mlsanity.com/paper/2608.04569/) (2026)
- [Protocol-Preserving Context Trimming for Agentic Workflows: Benefits, Failure Regimes, and Budget Guardrails](https://mlsanity.com/paper/2609.16461/) (2026)
- [MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression](https://mlsanity.com/paper/2609.27380/) (2026)
- [Adapting Context Compression for Long-Horizon Agents with Counterfactual Continuations](https://mlsanity.com/paper/2609.36526/) (2026)

## Abstract (arXiv)

The increasing prevalence of large language models (LLMs) such as GPT-4 in various applications has led to a surge in the size of prompts required for optimal performance, leading to challenges in computational efficiency. Prompt compression aims to reduce the inference cost by minimizing input tokens without compromising on the task performance. However, existing prompt compression techniques either rely on sub-optimal metrics such as information entropy or model it as a task-agnostic token classification problem that fails to capture task-specific information. To address these issues, we propose a novel and efficient reinforcement learning (RL) based task-aware prompt compression method. To ensure low latency requirements, we leverage existing Transformer encoder-based token classification model while guiding the learning process with task-specific reward signals using lightweight REINFORCE algorithm. We evaluate the performance of our method on three diverse and challenging tasks including text summarization, question answering and code summarization. We demonstrate that our RL-guided compression method improves the task performance by 8% - 189% across these three scenarios over state-of-the-art compression techniques while satisfying the same compression rate and latency requirements.