mlsanityData to 14 Oct 2026

DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training

arXiv 2605.18815 ↗First posted 12 May 2026cs.LG cs.DCcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Modern large language model (LLM) training is inherently dynamic: resource fluctuations, RLHF phase shifts, and cluster elasticity continually reshape the optimal parallelism layout, posing a significant challenge to existing training frameworks built around a static execution model. We present DynaTrain, a distributed training system for sub-second, online reconfiguration across arbitrary multi-dimensional parallelism. At its core, we propose a Virtual Parameter Space (VPS) abstraction that unifies all distributed training states under one logical coordinate space, turning any parallelism configuration into a deterministic mapping and collapsing complex transition into manageable geometric intersections. On top of VPS, a state routing-and-transition layer executes rank-local transfers under a memory-aware, deadlock-free schedule, and an Elastic Device Manager overlaps new-world construction with ongoing training to mask topology-change cost. On dense and MoE models up to 235B parameters, DynaTrain reconfigures a 70B dense model in under 2s and a 235B MoE model in 4.36s, outperforming state-of-the-art checkpoint-based and elastic systems by up to three orders of magnitude while preserving correctness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 3 of its 18 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

3 of 18 references.

References · 3

  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ReMP: Low-Downtime Runtime Model-Parallelism Reconfiguration for LLM Serving
  2. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  3. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
  4. 2025AsyncHZP: Hierarchical ZeRO Parallelism with Asynchronous Scheduling for Scalable LLM Training
  5. 2026FLYING SERVING: On-the-Fly Parallelism Switching for Large Language Model Serving
  6. 2026Heterogeneous Parallelism for Multimodal Large Language Model Training
  7. 2025Balanced and Elastic End-to-end Training of Dynamic LLMs
  8. 2025RAPID-LLM: Resilience-Aware Performance analysis of Infrastructure for Distributed LLM Training and Inference ↗