# GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment

arXiv: https://arxiv.org/abs/2505.12215 · HTML: https://mlsanity.com/paper/2505.12215/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving.
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 18 May 2025.

## Placed in

- Efficient LLM inference and serving / [Prompt and context compression](https://mlsanity.com/inference/prompt-and-context-compression/)

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Compressing Context to Enhance Inference Efficiency of Large Language Models](https://mlsanity.com/paper/2310.06201/) (2023)
- [Reducing Transformer Key-Value Cache Size with Cross-Layer Attention](https://mlsanity.com/paper/2405.12981/) (2024)
- [xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token](https://mlsanity.com/paper/2405.13792/) (2024)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [VoCo-LLaMA: Towards Vision Compression with Large Language Models](https://mlsanity.com/paper/2406.12275/) (2024)
- [CompAct: Compressing Retrieved Documents Actively for Question Answering](https://mlsanity.com/paper/2407.09014/) (2024)
- [500xCompressor: Generalized Prompt Compression for Large Language Models](https://mlsanity.com/paper/2408.03094/) (2024)
- [Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics](https://mlsanity.com/paper/2403.01509/) (2024)

## Built on by (most cited first, 21 of 21)

- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors](https://mlsanity.com/paper/2510.08907/) (2025)
- [COMI: Coarse-to-fine Context Compression via Marginal Information Gain](https://mlsanity.com/paper/2602.01719/) (2026)
- [Read As Human: Compressing Context via Parallelizable Close Reading and Skimming](https://mlsanity.com/paper/2602.01840/) (2026)
- [CoMeT: Collaborative Memory Transformer for Efficient Long Context Modeling](https://mlsanity.com/paper/2602.01766/) (2026)
- [SemantiCache: Efficient KV Cache Compression via Semantic Chunking and Clustered Merging](https://mlsanity.com/paper/2603.14303/) (2026)
- [CoS: Towards Optimal Event Scheduling via Chain-of-Scheduling](https://mlsanity.com/paper/2511.12913/) (2025)
- [End-to-End Context Compression at Scale](https://mlsanity.com/paper/2606.09659/) (2026)
- [ARC-Encoder: learning compressed text representations for large language models](https://mlsanity.com/paper/2510.20535/) (2025)
- [Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Driven](https://mlsanity.com/paper/2605.09463/) (2026)
- [Length-Adaptive Interest Network for Balancing Long and Short Sequence Modeling in CTR Prediction](https://mlsanity.com/paper/2601.19142/) (2026)
- [Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model](https://mlsanity.com/paper/2602.01778/) (2026)
- [Attn-GS: Attention-Guided Context Compression for Efficient Personalized LLMs](https://mlsanity.com/paper/2602.07778/) (2026)
- [MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration](https://mlsanity.com/paper/2604.14889/) (2026)
- [From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors](https://mlsanity.com/paper/2604.23277/) (2026)
- [SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems](https://mlsanity.com/paper/2606.03544/) (2026)
- [Thinking as Compression: Your Reasoning Model is Secretly a Context Compressor](https://mlsanity.com/paper/2605.28713/) (2026)
- [When Hard Negatives Hurt: Bridging the Generative-Discriminative Gap in Hard Negative Synthesis for Retrieval](https://mlsanity.com/paper/2606.01304/) (2026)
- [HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps](https://mlsanity.com/paper/2606.03768/) (2026)
- [Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAG](https://mlsanity.com/paper/2609.05152/) (2026)
- [Kwai Summary Attention V2 Technical Report](https://mlsanity.com/paper/2610.18654/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable performance across a wide range of Natural Language Processing (NLP) tasks. However, in long-context scenarios, they face two challenges: high computational cost and information redundancy. To address these challenges, we propose GMSA, an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. GMSA introduces Group Merging to achieve more uniform aggregation, mitigating semantic dominance during autoencoder pretraining, and Layer Semantic Alignment (LSA) to bridge the semantic gap between high-level abstract semantics and low-level input semantics. We first pretrain GMSA as an autoencoder and then fine-tune it for downstream tasks. Experiments demonstrate that GMSA improves context reconstruction compared to existing soft prompt compression paradigm and outperforms baselines on multiple long-context question answering and summarization benchmarks across two backbone models, while maintaining low end-to-end latency.