# Efficient Context Scaling with LongCat ZigZag Attention

arXiv: https://arxiv.org/abs/2512.23966 · HTML: https://mlsanity.com/paper/2512.23966/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 43 references that are in the map.
- First posted on arXiv 30 Dec 2025.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [YaRN: Efficient Context Window Extension of Large Language Models](https://mlsanity.com/paper/2309.00071/) (2023)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks](https://mlsanity.com/paper/2412.15204/) (2024)
- [DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads](https://mlsanity.com/paper/2410.10819/) (2024)
- [ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving](https://mlsanity.com/paper/2309.17452/) (2023)
- [HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly](https://mlsanity.com/paper/2410.02694/) (2024)
- [SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference](https://mlsanity.com/paper/2502.18137/) (2025)
- [Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries](https://mlsanity.com/paper/2409.12640/) (2024)
- [Long-Context Language Modeling with Parallel Context Encoding](https://mlsanity.com/paper/2402.16617/) (2024)
- [Evaluating Language Models for Efficient Code Generation](https://mlsanity.com/paper/2408.06450/) (2024)
- [On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models](https://mlsanity.com/paper/2512.07783/) (2025)
- [FullStack Bench: Evaluating LLMs as Full Stack Coders](https://mlsanity.com/paper/2412.00535/) (2024)
- [OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation](https://mlsanity.com/paper/2509.03498/) (2025)
- [Post-Training Sparse Attention with Double Sparsity](https://mlsanity.com/paper/2408.07092/) (2024)

## Built on by (most cited first, 6 of 6)

- [LongCat-Flash-Thinking-2601 Technical Report](https://mlsanity.com/paper/2601.16725/) (2026)
- [Arcee Trinity Large Technical Report](https://mlsanity.com/paper/2602.17004/) (2026)
- [Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers](https://mlsanity.com/paper/2601.17367/) (2026)
- [SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipelining](https://mlsanity.com/paper/2602.10718/) (2026)
- [Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference](https://mlsanity.com/paper/2604.07394/) (2026)
- [ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation](https://mlsanity.com/paper/2606.18056/) (2026)

## Abstract (arXiv)

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.