mlsanityData to 7 Oct 2026

TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction

arXiv 2310.15556 ↗First posted 24 Oct 2023cs.CL cs.IRcited by 34

On the map: Retrieval-augmented generation → Generation and context integration

also in 1 more
Abstract (arXiv)

Since ChatGPT released its API for public use, the number of applications built on top of commercial large language models (LLMs) increase exponentially. One popular usage of such models is leveraging its in-context learning ability and generating responses given user queries leveraging knowledge obtained by retrieval augmentation. One problem of deploying commercial retrieval-augmented LLMs is the cost due to the additionally retrieved context that largely increases the input token size of the LLMs. To mitigate this, we propose a token compression scheme that includes two methods: summarization compression and semantic compression. The first method applies a T5-based model that is fine-tuned by datasets generated using self-instruct containing samples with varying lengths and reduce token size by doing summarization. The second method further compresses the token size by removing words with lower impact on the semantic. In order to adequately evaluate the effectiveness of the proposed methods, we propose and utilize a dataset called Food-Recommendation DB (FRDB) focusing on food recommendation for women around pregnancy period or infants. Our summarization compression can reduce 65% of the retrieval token size with further 0.3% improvement on the accuracy; semantic compression provides a more flexible way to trade-off the token size with performance, for which we can reduce the token size by 20% with only 1.6% of accuracy drop.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 28 of its 46 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

28 of 46 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023In-Context Retrieval-Augmented Language Models31 Jan
  2. 2023REPLUG: Retrieval-Augmented Black-Box Language Models30 Jan
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Models5 Aug
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
12 more
  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  3. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  4. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  5. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  6. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  7. 2023A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3cited by 46
  8. 2021Improving Query Representations for Dense Retrieval with Pseudo Relevance Feedbackcited by 45
  9. 2023MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Promptingcited by 27
  10. 2023Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language modelcited by 14
  11. 2023Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A Practical Studycited by 5
  12. 2023CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Carecited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 14; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  2. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  3. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  4. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
  5. 2024Prompt Compression for Large Language Models: A Surveycited by 65
  6. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 46
  7. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  8. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  9. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  10. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 19
  11. 2024An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generationcited by 16
  12. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
22 more
  1. 2024MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoningcited by 12
  2. 2024A Hybrid RAG System with Comprehensive Enhancement on Complex Reasoningcited by 8
  3. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Modelscited by 8
  4. 2026Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inferencecited by 8
  5. 2024Urban Safety Perception Assessments via Integrating Multimodal Large Language Models with Street View Imagescited by 7
  6. 2024Enhancing Embedding Performance through Large Language Model-based Text Enrichment and Rewritingcited by 6
  7. 2025Key, Value, Compress: A Systematic Exploration of KV Cache Compression Techniquescited by 4
  8. 2025Lossless Token Sequence Compression via Meta-Tokenscited by 4
  9. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 4
  10. 2024Return of EM: Entity-driven Answer Set Expansion for QA Evaluationcited by 2
  11. 2024Not All Options Are Created Equal: Textual Option Weighting for Token-Efficient LLM-Based Knowledge Tracingcited by 2
  12. 2025Revisiting RAG Ensemble: A Theoretical and Mechanistic Analysis of Multi-RAG System Collaborationcited by 2
  13. 2026BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selectioncited by 2
  14. 2024Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Modelscited by 1
  15. 2025Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generationcited by 1
  16. 2025Efficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generationcited by 1
  17. 2025LLMs on a Budget? Say HOLAcited by 1
  18. 2024LLMBridge: Reducing Costs to Access LLMs in a Prompt-Centric Internetcited by 0
  19. 2025SlimRAG: Retrieval without Graphs via Entity-Aware Context Selectioncited by 0
  20. 2026SODA: Sensitivity-Oriented Dynamic Acceleration for Diffusion Transformercited by 0
  21. 2026Compressing Sequences in the Latent Embedding Space: K-Token Merging for Large Language Modelscited by 0
  22. 2026Relevant but Incomplete: Referential Dangling as a Paradigm-Level Failure Mode in Hard Prompt Compressioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction
  2. 2024500xCompressor: Generalized Prompt Compression for Large Language Models
  3. 2024Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs
  4. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  5. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  6. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
  7. 2025Learning to Compress: Unlocking the Potential of Large Language Models for Text Representation
  8. 2025AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation