mlsanityData to 7 Oct 2026

HiCI: Hierarchical Construction-Integration for Long-Context Attention

arXiv 2603.20843 ↗First posted 21 Mar 2026cs.CL cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Long-context language modeling is commonly framed as a scalability challenge of token-level attention, yet local-to-global information structuring remains largely implicit in existing approaches. Drawing on cognitive theories of discourse comprehension, we propose HiCI (Hierarchical Construction--Integration), a hierarchical attention module that constructs segment-level representations, integrates them into a shared global context, and broadcasts both to condition segment-level attention. We validate HiCI through parameter-efficient adaptation of LLaMA-2 with only <5.5% additional parameters, extending context from 4K to 100K tokens (7B) and 64K tokens (13B). Across language modeling, retrieval, and instruction-following benchmarks, HiCI yields consistent improvements over strong baselines, including matching proprietary models on topic retrieval and surpassing GPT-3.5-Turbo-16K on code comprehension. These results demonstrate the effectiveness of explicit hierarchical structuring as an inductive bias for long-context modeling.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 21 Mar 2026.

Builds on

45 of 63 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
33 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  8. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  9. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  10. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  11. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  12. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  13. 2022Recurrent Memory Transformercited by 197
  14. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  15. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  16. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 178
  17. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  18. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  19. 2022Block-Recurrent Transformerscited by 107
  20. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  21. 2024Large Concept Models: Language Modeling in a Sentence Representation Spacecited by 86
  22. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  23. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  24. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  25. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  26. 2020Deep Learning and the Global Workspace Theorycited by 35
  27. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inferencecited by 32
  28. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  29. 2025Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignmentcited by 11
  30. 2023Concept-Centric Transformers: Enhancing Model Interpretability through Object-Centric Concept Learning within a Shared Global Workspacecited by 7
  31. 2025Position: AI Scaling: From Up to Down and Outcited by 5
  32. 2025Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillationcited by 4
  33. 2023A Framework for Inference Inspired by Human Memory Mechanismscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  2. 2026Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recoverycited by 4