mlsanityData to 7 Oct 2026

MinT: Managed Infrastructure for Training and Serving Millions of LLMs

arXiv 2605.13779 ↗First posted 13 May 2026cs.LG cs.AI cs.DCcited by 1

On the map: Mixture-of-experts → Mixture of LoRA experts

Abstract (arXiv)

We present MindLab Toolkit (MinT), a managed infrastructure system for Low-Rank Adaptation (LoRA) post-training and online serving. MinT targets a setting where many trained policies are produced over a small number of expensive base-model deployments. Instead of materializing each policy as a merged full checkpoint, MinT keeps the base model resident and moves exported LoRA adapter revisions through rollout, update, export, evaluation, serving, and rollback, hiding distributed training, serving, scheduling, and data movement behind a service interface. MinT scales this path along three axes. Scale Up extends LoRA RL to frontier-scale dense and MoE architectures, including MLA and DSA attention paths, with training and serving validated beyond 1T total parameters. Scale Down moves only the exported LoRA adapter, which can be under 1% of base-model size in rank-1 settings; adapter-only handoff reduces the measured step by 18.3x on a 4B dense model and 2.85x on a 30B MoE, while concurrent multi-policy GRPO shortens wall time by 1.77x and 1.45x without raising peak memory. Scale Out separates durable policy addressability from CPU/GPU working sets: a tensor-parallel deployment supports 10^6-scale addressable catalogs (measured single-engine sweeps through 100K) and thousand-adapter active waves at cluster scale, with cold loading treated as scheduled service work and packed MoE LoRA tensors improving live engine loading by 8.5-8.7x. MinT thus manages million-scale LoRA policy catalogs while training and serving selected adapter revisions over shared 1T-class base models.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Mixture of LoRA experts, one of 12 directions in Mixture-of-experts. Builds on 29 of its 48 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

29 of 48 references.

References · 29

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  7. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  8. 2025Kimi K2: Open Agentic Intelligencecited by 523
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  10. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  11. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  12. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
17 more
  1. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  2. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
  3. 2023Punica: Multi-Tenant LoRA Servingcited by 80
  4. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  5. 2023FinGPT: Democratizing Internet-scale Data for Financial Large Language Modelscited by 65
  6. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  7. 2023FinEval: A Chinese Financial Domain Knowledge Evaluation Benchmark for Large Language Modelscited by 52
  8. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  9. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  10. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  11. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  12. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  13. 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 14
  14. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  15. 2026Jet-RL: Enabling On-Policy FP8 Reinforcement Learning with Unified Training and Rollout Precision Flowcited by 11
  16. 2025Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Managementcited by 4
  17. 2025Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systemscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026JoyNexus: Service-Oriented Multi-Tenant Post-Training for VLA Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRALib: A Standardized Benchmark for Evaluating LoRA-MoE Methods
  2. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices
  3. 2025Activated LoRA: Fine-tuned LLMs for Intrinsics
  4. 2025Predictive-LoRA: A Proactive and Fragmentation-Aware Serverless Inference System for LLMs
  5. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  6. 2026Co-occurrence Patterns of LoRA Adapters in Production Diffusion Model Inference Services
  7. 2026Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
  8. 2025MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment