mlsanityData to 7 Oct 2026

GPT Carry-On: Training Foundation Model for Customization Could Be Simple, Scalable and Affordable

arXiv 2504.07513 ↗First posted 10 Apr 2025cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern large language foundation models (LLM) have now entered the daily lives of millions of users. We ask a natural question whether it is possible to customize LLM for every user or every task. From system and industrial economy consideration, general continue-training or fine-tuning still require substantial computation and memory of training GPU nodes, whereas most inference nodes under deployment, possibly with lower-end GPUs, are configured to make forward pass fastest possible. We propose a framework to take full advantages of existing LLMs and systems of online service. We train an additional branch of transformer blocks on the final-layer embedding of pretrained LLMs, which is the base, then a carry-on module merge the base models to compose a customized LLM. We can mix multiple layers, or multiple LLMs specialized in different domains such as chat, coding, math, to form a new mixture of LLM that best fit a new task. As the base model don't need to update parameters, we are able to outsource most computation of the training job on inference nodes, and only train a lightweight carry-on on training nodes, where we consume less than 1GB GPU memory to train a 100M carry-on layer on 30B LLM. We tested Qwen and DeepSeek opensourced models for continue-pretraining and got faster loss convergence. We use it to improve solving math questions with extremely small computation and model size, with 1000 data samples of chain-of-thoughts, and as small as 1 MB parameters of two layer layer carry-on, and the results are promising.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 21 references that are in the map. First posted on arXiv 10 Apr 2025.

Builds on

14 of 21 references.

References · 14

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2022Efficiently Scaling Transformer Inferencecited by 529
2 more
  1. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  2. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  2. 2025Efficient Multi-Model Orchestration for Self-Hosted Large Language Models
  3. 2026Lodestar: An Online-Learning LLM Inference Router
  4. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuning
  5. 2023Confidant: Customizing Transformer-based LLMs via Collaborative Edge Training
  6. 2025Harli: SLO-Aware Co-location of LLM Inference and PEFT-based Finetuning on Model-as-a-Service Platforms
  7. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
  8. 2023Efficient LLM inference solution on Intel GPU