mlsanityData to 7 Oct 2026

Arcee Trinity Large Technical Report

arXiv 2602.17004 ↗First posted 19 Feb 2026cs.LG cs.CLcited by 10

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 43 of its 61 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

43 of 61 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
31 more
  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  7. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  8. 2024Measuring short-form factuality in large language modelscited by 345
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  11. 2020Query-Key Normalization for Transformerscited by 266
  12. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  13. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  14. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  15. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  16. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  17. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  18. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 123
  19. 2025olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Modelscited by 120
  20. 2024Tokenization counts: the impact of tokenization on arithmetic in frontier LLMscited by 112
  21. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 80
  22. 2025Dion: Distributed Orthonormalized Updatescited by 64
  23. 2025SuperBPE: Space Travel for Language Modelscited by 58
  24. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 52
  25. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 51
  26. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 51
  27. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  28. 2024Cut Your Losses in Large-Vocabulary Language Modelscited by 35
  29. 2025Peri-LN: Revisiting Normalization Layer in the Transformer Architecturecited by 32
  30. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretrainingcited by 19
  31. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
  2. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Datacited by 6
  3. 2026Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routerscited by 5
  4. 2026Anytime Training with Schedule-Free Spectral Optimizationcited by 3
  5. 2026Aurora: A Leverage-Aware Spectral Optimizercited by 3
  6. 2026Full-bandwidth transformercited by 3
  7. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcyclingcited by 2
  8. 2026Mellum2 Technical Reportcited by 0
  9. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Expertscited by 0
  10. 2026Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training
  2. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  3. 2026LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale
  5. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  6. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  7. 2025LongCat-Flash Technical Report
  8. 2026Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts