360-LLaMA-Factory: Plug & Play Sequence Parallelism for Long Post-Training
Not on the map. Why?
Abstract (arXiv)
Adding sequence parallelism into LLaMA-Factory, we open-sourced 360-LLaMA-Factory at https://github.com/Qihoo360/360-LLaMA-Factory. 360-LLaMA-Factory has received wide recognition and used in models such as Light-R1 arXiv:2503.10460, TinyR1 arXiv:2503.04872, Kaggle AIMO math models and also in large companies' training frameworks. This technical report delves deeper into the different sequence parallel modes behind 360-LLaMA-Factory and discusses our implementation insights.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 11 of its 18 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
11 of 18 references.
References · 11
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
- 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
- 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
- 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
- 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 58
- 2024LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context Parallelismcited by 16
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.