# Core Context Aware Transformers for Long Context Language Modeling

arXiv: https://arxiv.org/abs/2412.12465 · HTML: https://mlsanity.com/paper/2412.12465/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Dec 2024.

## Built on by (most cited first, 11 of 11)

- [Long-Context Generalization with Sparse Attention](https://mlsanity.com/paper/2506.16640/) (2025)
- [FSA: An Alternative Efficient Implementation of Native Sparse Attention Kernel](https://mlsanity.com/paper/2508.18224/) (2025)
- [Curse of High Dimensionality Issue in Transformer for Long-context Modeling](https://mlsanity.com/paper/2505.22107/) (2025)
- [Towards Efficient Agents: A Co-Design of Inference Architecture and System](https://mlsanity.com/paper/2512.18337/) (2025)
- [Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention](https://mlsanity.com/paper/2603.08743/) (2026)
- [EvidFuse: Writing-Time Evidence Learning for Consistent Text-Chart Data Reporting](https://mlsanity.com/paper/2601.05487/) (2026)
- [Training-free Context-adaptive Attention for Efficient Long Context Modeling](https://mlsanity.com/paper/2512.09238/) (2025)
- [S3-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference](https://mlsanity.com/paper/2601.17702/) (2026)
- [Training-Inference Consistent Segmented Execution for Long-Context LLMs](https://mlsanity.com/paper/2605.11744/) (2026)
- [Latent-Condensed Transformer for Efficient Long Context Modeling](https://mlsanity.com/paper/2604.12452/) (2026)
- [LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining](https://mlsanity.com/paper/2608.12419/) (2026)

## Abstract (arXiv)

Transformer-based Large Language Models (LLMs) have exhibited remarkable success in extensive tasks primarily attributed to self-attention mechanism, which requires a token to consider all preceding tokens as its context to compute attention. However, when the context length L becomes very large (e.g., 128K), the amount of potentially redundant information in the context tends to increase. The redundant context not only hampers the modeling representation performance but also incurs unnecessary computational and storage overhead. In this paper, we propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling, comprising two complementary modules: 1) Globality-aware pooling module groups input tokens and dynamically compresses each group into one core token based on their significance. In this way, our method automatically focuses and strengthens core context while diminishing redundancy during the learning process, leading to effective long-term dependency modeling. 2) Locality-preserving module incorporates neighboring tokens to preserve local context for detailed representation. Notably, our CCA-Attention is able to replace the self-attention module in existing LLMs with minimal fine-tuning cost. Extensive experimental results show the superiority of our method in both long-context modeling and computational efficiency over state-of-the-art methods.