mlsanityData to 14 Oct 2026

Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought

arXiv 2505.15431 ↗First posted 21 May 2025cs.CLcited by 29

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 2 more
Abstract (arXiv)

As Large Language Models (LLMs) rapidly advance, we introduce Hunyuan-TurboS, a novel large hybrid Transformer-Mamba Mixture of Experts (MoE) model. It synergistically combines Mamba's long-sequence processing efficiency with Transformer's superior contextual understanding. Hunyuan-TurboS features an adaptive long-short chain-of-thought (CoT) mechanism, dynamically switching between rapid responses for simple queries and deep "thinking" modes for complex problems, optimizing computational resources. Architecturally, this 56B activated (560B total) parameter model employs 128 layers (Mamba2, Attention, FFN) with an innovative AMF/MF block pattern. Faster Mamba2 ensures linear complexity, Grouped-Query Attention minimizes KV cache, and FFNs use an MoE structure. Pre-trained on 16T high-quality tokens, it supports a 256K context length and is the first industry-deployed large-scale Mamba model. Our comprehensive post-training strategy enhances capabilities via Supervised Fine-Tuning (3M instructions), a novel Adaptive Long-short CoT Fusion method, Multi-round Deliberation Learning for iterative improvement, and a two-stage Large-scale Reinforcement Learning process targeting STEM and general instruction-following. Evaluations show strong performance: overall top 7 rank on LMSYS Chatbot Arena with a score of 1356, outperforming leading models like Gemini-2.0-Flash-001 (1352) and o4-mini-2025-04-16 (1345). TurboS also achieves an average of 77.9% across 23 automated benchmarks. Hunyuan-TurboS balances high performance and efficiency, offering substantial capabilities at lower inference costs than many reasoning models, establishing a new paradigm for efficient large-scale pre-trained models.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 37 of its 59 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

37 of 59 references.

References · 37

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
25 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  4. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  5. 2024DeepSeek-V3 Technical Reportcited by 811
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  10. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  11. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  12. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 256
  13. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  14. 2024Are We Done with MMLU?cited by 194
  15. 2023Self-Alignment with Instruction Backtranslationcited by 151
  16. 2024Generative Reward Modelscited by 148
  17. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  18. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
  19. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
  20. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
  21. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  22. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimizationcited by 21
  23. 2021CCPM: A Chinese Classical Poetry Matching Datasetcited by 15
  24. 2024Halu-J: Critique-Based Hallucination Judgecited by 14
  25. 2023Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencentcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 12.

29 papers.

Sort:

By area:

  1. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  4. 2025ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluationcited by 42
  5. 2025AMO-Bench: Large Language Models Still Struggle in High School Math Competitionscited by 38
  6. 2025A Survey on LLM Mid-Trainingcited by 21
  7. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  8. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  9. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  10. 2025Scaling Linear Attention with Sparse State Expansioncited by 13
  11. 2025Reinforcement Learning on Pre-Training Datacited by 12
  12. 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 11
17 more
  1. 2025Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMscited by 10
  2. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  3. 2025VACoT: Rethinking Visual Data Augmentation with VLMscited by 9
  4. 2025LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Servicescited by 8
  5. 2026MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM-Powered Assistantscited by 8
  6. 2025Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Modelscited by 6
  7. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Taskscited by 6
  8. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  9. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  10. 2026A Provable Expressiveness Hierarchy in Hybrid Linear-Full Attentioncited by 1
  11. 2026Towards Faithful Industrial RAG: A Reinforced Co-adaptation Framework for Advertising QAcited by 1
  12. 2026Human or Machine? A Preliminary Turing Test for Speech-to-Speech Interactioncited by 1
  13. 2025Streaming Tensor Programs: A Streaming Abstraction for Dynamic Parallelismcited by 0
  14. 2026Architecture-Aware Reinforcement Learning Makes Sliding-Window Attention Competitive in Math Reasoningcited by 0
  15. 2026NOVA: Technology-Architecture Co-Design of Near-Memory Processing for Attention-SSM-MoE Hybrid LLM Inferencecited by 0
  16. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignmentcited by 0
  17. 2026Hunyuan-A13B Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models
  2. 2024An Empirical Study of Mamba-based Language Models
  3. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  4. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  5. 2024BlackMamba: Mixture of Experts for State-Space Models
  6. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  7. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  8. 2024Taipan: Efficient and Expressive State Space Language Models with Selective Attention