mlsanityData to 7 Oct 2026

Aurora:Activating Chinese chat capability for Mixtral-8x7B sparse Mixture-of-Experts through Instruction-Tuning

arXiv 2312.14557 ↗First posted 22 Dec 2023cs.CLcited by 3

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Existing research has demonstrated that refining large language models (LLMs) through the utilization of machine-generated instruction-following data empowers these models to exhibit impressive zero-shot capabilities for novel tasks, without requiring human-authored instructions. In this paper, we systematically investigate, preprocess, and integrate three Chinese instruction-following datasets with the aim of enhancing the Chinese conversational capabilities of Mixtral-8x7B sparse Mixture-of-Experts model. Through instruction fine-tuning on this carefully processed dataset, we successfully construct the Mixtral-8x7B sparse Mixture-of-Experts model named "Aurora." To assess the performance of Aurora, we utilize three widely recognized benchmark tests: C-Eval, MMLU, and CMMLU. Empirical studies validate the effectiveness of instruction fine-tuning applied to Mixtral-8x7B sparse Mixture-of-Experts model. This work is pioneering in the execution of instruction fine-tuning on a sparse expert-mixed model, marking a significant breakthrough in enhancing the capabilities of this model architecture. Our code, data and model are publicly available at https://github.com/WangRongsheng/Aurora

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 9 of its 12 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

9 of 12 references.

References · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  9. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  2. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  3. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12

Similar, not cited

Close in wording, no citation either way

  1. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models
  2. 2023An Empirical Study of Instruction-tuning Large Language Models in Chinese
  3. 2024Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral
  4. 2024Mixtral of Experts
  5. 2024Mixture-of-Instructions: Aligning Large Language Models via Mixture Prompting
  6. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Experts
  7. 2024Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks
  8. 2026Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling