# Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs

arXiv: https://arxiv.org/abs/2404.10308 · HTML: https://mlsanity.com/paper/2404.10308/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [QuALITY: Question Answering with Long Input Texts, Yes!](https://mlsanity.com/paper/2112.08608/) (2021)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [Blockwise Self-Attention for Long Document Understanding](https://mlsanity.com/paper/1911.02972/) (2019)
- [Unlimiformer: Long-Range Transformers with Unlimited Length Input](https://mlsanity.com/paper/2305.01625/) (2023)
- [Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?](https://mlsanity.com/paper/2207.10551/) (2022)
- [Efficient Long-Text Understanding with Short-Text Models](https://mlsanity.com/paper/2208.00748/) (2022)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)

## Built on by (most cited first, 30 of 30)

- [In-Context Learning with Long-Context Models: An In-Depth Exploration](https://mlsanity.com/paper/2405.00200/) (2024)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding](https://mlsanity.com/paper/2505.20862/) (2025)
- [HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processing](https://mlsanity.com/paper/2405.06067/) (2024)
- [Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models](https://mlsanity.com/paper/2410.16983/) (2024)
- [SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation](https://mlsanity.com/paper/2412.13649/) (2024)
- [Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning](https://mlsanity.com/paper/2406.02547/) (2024)
- [In-Context Former: Lightning-fast Compressing Context for Large Language Model](https://mlsanity.com/paper/2406.13618/) (2024)
- [EVICPRESS: Joint KV-Cache Compression and Eviction for Efficient LLM Serving](https://mlsanity.com/paper/2512.14946/) (2025)
- [Context-Aware Hierarchical Merging for Long Document Summarization](https://mlsanity.com/paper/2502.00977/) (2025)
- [A Simple yet Effective Layout Token in Large Language Models for Document Understanding](https://mlsanity.com/paper/2503.18434/) (2025)
- [Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models](https://mlsanity.com/paper/2505.20873/) (2025)
- [FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference](https://mlsanity.com/paper/2601.13143/) (2026)
- [Gated Differentiable Working Memory for Long-Context Language Modeling](https://mlsanity.com/paper/2601.12906/) (2026)
- [Large Language Model as Token Compressor and Decompressor](https://mlsanity.com/paper/2603.25340/) (2026)
- [Evaluating LLaMA 3.2 for Software Vulnerability Detection](https://mlsanity.com/paper/2503.07770/) (2025)
- [Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Models](https://mlsanity.com/paper/2506.00773/) (2025)
- [Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers](https://mlsanity.com/paper/2506.01215/) (2025)
- [Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives?](https://mlsanity.com/paper/2512.07777/) (2025)
- [Boosting Long-Context Management via Query-Guided Activation Refilling](https://mlsanity.com/paper/2412.12486/) (2024)
- [StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation](https://mlsanity.com/paper/2506.04567/) (2025)
- [A novel language model for predicting serious adverse event results in clinical trials from their prospective registrations](https://mlsanity.com/paper/2507.22919/) (2025)
- [Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding](https://mlsanity.com/paper/2602.02977/) (2026)
- [Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity](https://mlsanity.com/paper/2605.22476/) (2026)
- [Sparsified State-Space Models are Efficient Highway Networks](https://mlsanity.com/paper/2505.20698/) (2025)
- [Zonkey: A Hierarchical Diffusion Language Model with Differentiable Tokenization and Probabilistic Attention](https://mlsanity.com/paper/2601.21768/) (2026)
- [Make Your LVLM KV Cache More Lightweight](https://mlsanity.com/paper/2605.00789/) (2026)
- [Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving](https://mlsanity.com/paper/2607.05399/) (2026)
- [Contextualized Early Detection of Online Firestorms: A Sequential LLM-Based Approach](https://mlsanity.com/paper/2607.14957/) (2026)
- [RH-RAG: Trustworthy Long-Form Generation for Privacy-Constrained Settings](https://mlsanity.com/paper/2608.01311/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable performance in various natural language processing tasks. However, a primary constraint they face is the context limit, i.e., the maximum number of tokens they can process. Previous works have explored architectural changes and modifications in positional encoding to relax the constraint, but they often require expensive training or do not address the computational demands of self-attention. In this paper, we present Hierarchical cOntext MERging (HOMER), a new training-free scheme designed to overcome the limitations. HOMER uses a divide-and-conquer algorithm, dividing long inputs into manageable chunks. Each chunk is then processed collectively, employing a hierarchical strategy that merges adjacent chunks at progressive transformer layers. A token reduction technique precedes each merging, ensuring memory usage efficiency. We also propose an optimized computational order reducing the memory requirement to logarithmically scale with respect to input length, making it especially favorable for environments with tight memory restrictions. Our experiments demonstrate the proposed method's superior performance and memory efficiency, enabling the broader use of LLMs in contexts requiring extended context. Code is available at https://github.com/alinlab/HOMER.