mlsanityData to 14 Oct 2026

Efficient Context Scaling with LongCat ZigZag Attention

arXiv 2512.23966 ↗First posted 30 Dec 2025cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 38 of its 43 references that are in the map. First posted on arXiv 30 Dec 2025.

Builds on

38 of 43 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
26 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  2. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  6. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  7. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  9. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  10. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  11. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 120
  12. 2024Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queriescited by 88
  13. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  14. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  15. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  16. 2024FullStack Bench: Evaluating LLMs as Full Stack Coderscited by 58
  17. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  18. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  19. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  20. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 32
  21. 2025A Survey on LLM Mid-Trainingcited by 21
  22. 2025GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agentscited by 14
  23. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 12
  24. 2025Rectified Sparse Attentioncited by 10
  25. 2025FlashMLA-ETAP: Efficient Transpose Attention Pipeline for Accelerating MLA Inference on NVIDIA H20 GPUscited by 5
  26. 2025ZigzagAttention: Efficient Long-Context Inference with Exclusive Retrieval and Streaming Headscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  2. 2026Arcee Trinity Large Technical Reportcited by 11
  3. 2026Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformerscited by 5
  4. 2026SnapMLA: Efficient Long-Context MLA Decoding via Hardware-Aware FP8 Quantized Pipeliningcited by 4
  5. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inferencecited by 0
  6. 2026ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Kascade: A Practical Sparse Attention Method for Long-Context LLM Inference
  2. 2026LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning
  3. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
  4. 2026SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
  5. 2025Stream: Scaling up Mechanistic Interpretability to Long Context in LLMs via Sparse Attention
  6. 2025Optimizing Native Sparse Attention with Latent Attention and Local Global Alternating Strategies
  7. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
  8. 2026You Only Index Once: Cross-Layer Sparse Attention with Shared Routing