mlsanityData to 7 Oct 2026

ZAYA1-8B Technical Report

arXiv 2605.05365 ↗First posted 6 May 2026cs.AI cs.CLcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining, midtraining, and supervised fine-tuning (SFT) were performed on a full-stack AMD compute, networking, and software platform. With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, and remains competitive with substantially larger open-weight reasoning models. ZAYA1-8B was trained from scratch for reasoning, with reasoning data included from pretraining onward using an answer-preserving trimming scheme. Post-training uses a four-stage RL cascade: reasoning warmup on math and puzzles; a 400-task RLVE-Gym curriculum; math and code RL with test-time compute traces and synthetic code environments built from competitive-programming references; and behavioral RL for chat and instruction following. We also introduce Markovian RSA, a test-time compute method that recursively aggregates parallel reasoning traces while carrying forward only bounded-length reasoning tails between rounds. In TTC evaluation, Markovian RSA raises ZAYA1-8B to 91.9% on AIME'25 and 89.6% on HMMT'25 while carrying forward only a 4K-token tail, narrowing the gap to much larger reasoning models including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 41 of its 58 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

41 of 58 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning9 Jan
  2. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning23 Dec
  3. 2025Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models30 Sep
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention16 Jun
  5. 2025Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning5 Jun
  6. 2025Shorten After You're Right: Lazy Length Penalties for Reasoning RL18 May

Other references · 35

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
23 more
  1. 2025Muon is Scalable for LLM Trainingcited by 381
  2. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  3. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  4. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  5. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
  6. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  7. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  8. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  9. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  10. 2025Defeating the Training-Inference Mismatch via FP16cited by 62
  11. 2025CodeI/O: Condensing Reasoning Patterns via Code Input-Output Predictioncited by 54
  12. 2024DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averagingcited by 48
  13. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  14. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 44
  15. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  16. 2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
  17. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 29
  18. 2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 26
  19. 2025CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesiscited by 15
  20. 2026Training Language Models via Neural Cellular Automatacited by 15
  21. 2026Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMscited by 6
  22. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Spacecited by 5
  23. 2025Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Designcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  2. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 1
  3. 2026Scaling Adaptive Depth with Norm-Agnostic Residual Networkscited by 0
  4. 2026Rufus-Air: An Open LLM Post-Training Recipecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMs
  2. 2025Holistic Capability Preservation: Towards Compact Yet Comprehensive Reasoning Models
  3. 2025Skill-Based Mixture-of-Experts: Adaptive Routing for Heterogeneous Reasoning via Inferred Skills
  4. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
  5. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Models
  6. 2026Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition
  7. 2025Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
  8. 2026Post-Trained MoE Can Skip Half Experts via Self-Distillation