mlsanityData to 7 Oct 2026

H2MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer

arXiv 2605.24930 ↗First posted 24 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformer-based LLMs achieve strong results on many language tasks; however, long inputs remain challenging because context windows are finite, and prefill latency and memory grow rapidly with prompt length. Flat token-stream processing and chunk-based retrieval can therefore spend substantial computation and context budget on text unrelated to the query. Offline-indexed RAG additionally introduces external storage and index management overhead, and typically appends retrieved evidence as raw text, increasing prefill cost and latency. H^{2}MT makes long-context inference structure-aware: it builds a semantic hierarchy offline, computes a memory embedding for each node via bottom-up post-order aggregation, and routes queries coarse-to-fine at inference to prune irrelevant branches early. On LongBench QA (NarrativeQA, HotpotQA, QASPER) and two structured technical-document settings, H MT achieves favorable quality efficiency trade-offs, delivering competitive ROUGE-L and F1 (where applicable) with lower peak GPU memory and time-to-first-token (TTFT) than prompt compression, memory-token methods, and retrieval-augmented generation baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 24 May 2026.

Builds on

18 of 29 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  9. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  10. 2023Adapting Language Models to Compress Contextscited by 350
  11. 2022Memorizing Transformerscited by 219
  12. 2022Recurrent Memory Transformercited by 197
6 more
  1. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  2. 2024Long Context Compression with Activation Beaconcited by 101
  3. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  4. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  5. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  6. 2026IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding
  2. 2024Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction
  3. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference
  4. 2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs
  5. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  6. 2025HA-RAG: Hotness-Aware RAG Acceleration via Mixed Precision and Data Placement
  7. 2026LycheeCluster: Efficient Long-Context Inference with Structure-Aware Chunking and Hierarchical KV Indexing
  8. 2026BEAVER: A Training-Free Hierarchical Prompt Compression Method via Structure-Aware Page Selection