mlsanityData to 14 Oct 2026

Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

arXiv 2411.02265 ↗First posted 4 Nov 2024cs.CL cs.AIcited by 90

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 51 of its 59 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

51 of 59 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024HMoE: Heterogeneous Mixture of Experts for Language Modeling20 Aug
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  3. 2024Mixtral of Experts8 Jan
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
35 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  13. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  14. 2020GLU Variants Improve Transformercited by 1,804
  15. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  16. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  17. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  18. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  19. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  20. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  21. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  22. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  23. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  24. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  25. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  26. 2024LLM Critics Help Catch LLM Bugscited by 147
  27. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  28. 2024Nemotron-4 340B Technical Reportcited by 144
  29. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  30. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
  31. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 81
  32. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  33. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  34. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  35. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 58; 2026: 28.

90 papers.

Sort:

By area:

  1. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  2. 2025From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeerscited by 136
  3. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  4. 2025Scaling Laws for Native Multimodal Modelscited by 50
  5. 2025Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Modelscited by 45
  6. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thoughtcited by 29
  7. 2025SpeCa: Accelerating Diffusion Transformers with Speculative Feature Cachingcited by 29
  8. 2025Forecast then Calibrate: Feature Caching as ODE for Efficient Diffusion Transformerscited by 26
  9. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 24
  10. 2024SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecuritycited by 21
  11. 2025Mid-Training of Large Language Models: A Surveycited by 21
  12. 2025A Survey on LLM Mid-Trainingcited by 21
38 more
  1. 2024Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimizationcited by 20
  2. 2025Compute Only 16 Tokens in One Timestep: Accelerating Diffusion Transformers with Cluster-Driven Feature Cachingcited by 19
  3. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  4. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 15
  5. 2025Pandora3D: A Comprehensive Framework for High-Quality 3D Shape and Texture Generationcited by 14
  6. 2025Scaling Laws for Upcycling Mixture-of-Experts Language Modelscited by 13
  7. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missionscited by 12
  8. 2025Pangu Ultra MoE: How to Train Your Big MoE on Ascend NPUscited by 12
  9. 2026Light Forcing: Accelerating Autoregressive Video Diffusion via Sparse Attentioncited by 12
  10. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  11. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Expertscited by 10
  12. 2025T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitratescited by 10
  13. 2025Dynamic Expert Specialization: Towards Catastrophic Forgetting-Free Multi-Domain MoE Adaptationcited by 10
  14. 2025Towards a Comprehensive Scaling Law of Mixture-of-Expertscited by 10
  15. 2025Let Features Decide Their Own Solvers: Hybrid Feature Caching for Diffusion Transformerscited by 10
  16. 2025LLM-Enhanced Dialogue Management for Full-Duplex Spoken Dialogue Systemscited by 9
  17. 2025OmniCam: Unified Multimodal Video Generation via Camera Controlcited by 9
  18. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
  19. 2024Scaling Laws for Black box Adversarial Attackscited by 8
  20. 2025Scaling Laws for Floating Point Quantization Trainingcited by 8
  21. 2025SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generationcited by 8
  22. 2025RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Contextcited by 8
  23. 2025DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animationcited by 7
  24. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
  25. 2026S2DiT: Sandwich Diffusion Transformer for Mobile Streaming Video Generationcited by 7
  26. 2026DisCa: Accelerating Video Diffusion Transformers with Distillation-Compatible Learnable Feature Cachingcited by 7
  27. 2026SongBench: A Fine-Grained Multi-Aspect Benchmark for Song Quality Assessmentcited by 7
  28. 2025AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Fieldcited by 6
  29. 2025Patterns behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inferencecited by 6
  30. 2026From Sketch to Fresco: Efficient Diffusion Transformer with Progressive Resolutioncited by 6
  31. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  32. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 6
  33. 2025Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysiscited by 5
  34. 2025T2VAttack: Adversarial Attack on Text-to-Video Diffusion Modelscited by 5
  35. 2026PhyEduVideo: A Benchmark for Evaluating Text-to-Video Models for Physics Educationcited by 5
  36. 2025Temporal In-Context Fine-Tuning with Temporal Reasoning for Versatile Control of Video Diffusion Modelscited by 4
  37. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 4
  38. 2026Forecast the Principal, Stabilize the Residual: Subspace-Aware Feature Caching for Efficient Diffusion Transformerscited by 4

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Experts in Large Language Models
  2. 2021Scalable and Efficient MoE Training for Multitask Multilingual Models
  3. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  4. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  5. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
  7. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  8. 2025Faster MoE LLM Inference for Extremely Large Models