mlsanityData to 7 Oct 2026

SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning

arXiv 2408.05517 ↗First posted 10 Aug 2024cs.CLcited by 345

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent development in Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs) have leverage Attention-based Transformer architectures and achieved superior performance and generalization capabilities. They have since covered extensive areas of traditional learning tasks. For instance, text-based tasks such as text-classification and sequence-labeling, as well as multi-modal tasks like Visual Question Answering (VQA) and Optical Character Recognition (OCR), which were previously addressed using different models, can now be tackled based on one foundation model. Consequently, the training and lightweight fine-tuning of LLMs and MLLMs, especially those based on Transformer architecture, has become particularly important. In recognition of these overwhelming needs, we develop SWIFT, a customizable one-stop infrastructure for large models. With support of over 300+ LLMs and 50+ MLLMs, SWIFT stands as the open-source framework that provide the most comprehensive support for fine-tuning large models. In particular, it is the first training framework that provides systematic support for MLLMs. In addition to the core functionalities of fine-tuning, SWIFT also integrates post-training processes such as inference, evaluation, and model quantization, to facilitate fast adoptions of large models in various application scenarios. With a systematic integration of various training techniques, SWIFT offers helpful utilities such as benchmark comparisons among different training techniques for large models. For fine-tuning models specialized in agent framework, we show that notable improvements on the ToolBench leader-board can be achieved by training with customized dataset on SWIFT, with an increase of 5.2%-21.8% in the Act.EM metric over various baseline models, a reduction in hallucination by 1.6%-14.1%, and an average performance improvement of 8%-17%.

Key facts. Cited by 345 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 59 references that are in the map. First posted on arXiv 10 Aug 2024.

Builds on

31 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
18 more
  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2021GPT Understands, Toocited by 715
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  8. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  9. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
  10. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  11. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  12. 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 97
  13. 2021M6: A Chinese Multimodal Pretrainercited by 91
  14. 2024Parameter-Efficient Fine-Tuning with Discrete Fourier Transformcited by 67
  15. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 58
  16. 2023SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editingcited by 37
  17. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  18. 2023Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbonecited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 122; 2026: 212.

345 papers.

Sort:

By area:

  1. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  2. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  3. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  4. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  5. 2025Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Modelscited by 94
  6. 2024Text4Seg: Reimagining Image Segmentation as Text Generationcited by 59
  7. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgetingcited by 50
  8. 2025DeepAnalyze: Agentic Large Language Models for Autonomous Data Sciencecited by 49
  9. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  10. 2025Advancing Multimodal Reasoning via Reinforcement Learning with Cold Startcited by 41
  11. 2024Sparkle: Mastering Basic Spatial Capabilities in Vision Language Models Elicits Generalization to Spatial Reasoningcited by 33
  12. 2025Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoningcited by 23
38 more
  1. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  2. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21
  3. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  4. 2024CHOICE: Benchmarking the Remote Sensing Capabilities of Large Vision-Language Modelscited by 19
  5. 2025Can Pruning Improve Reasoning? Revisiting Long-CoT Compression with Capability in Mind for Better Reasoningcited by 19
  6. 2025Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learningcited by 19
  7. 2025RobustFlow: Towards Robust Agentic Workflow Generationcited by 19
  8. 2025Seeing Before Reasoning: A Unified Framework for Generalizable and Explainable Fake Image Detectioncited by 19
  9. 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 18
  10. 2025HumanAesExpert: Advancing a Multi-Modality Foundation Model for Human Image Aesthetic Assessmentcited by 17
  11. 2025AWorld: Orchestrating the Training Recipe for Agentic AIcited by 16
  12. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  13. 2026Qwen-Image-Bench: From Generation to Creation in Text-to-Image Evaluationcited by 16
  14. 2024From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approachcited by 15
  15. 2025WavReward: Spoken Dialogue Models With Generalist Reward Evaluatorscited by 15
  16. 2025SEEA-R1: Tree-Structured Reinforcement Fine-Tuning for Self-Evolving Embodied Agentscited by 15
  17. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  18. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
  19. 2025Revisiting Visual Understanding in Multimodal Reasoning through a Lens of Image Perturbationcited by 14
  20. 2026Automated Optimization Modeling via a Localizable Error-Driven Perspectivecited by 14
  21. 2026DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generationcited by 14
  22. 2026Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservationcited by 14
  23. 2024Automatic Evaluation for Text-to-image Generation: Task-decomposed Framework, Distilled Training, and Meta-evaluation Benchmarkcited by 13
  24. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  25. 2025VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generationcited by 13
  26. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Predictioncited by 13
  27. 2025ConCuR: Conciseness Makes State-of-the-Art Kernel Generationcited by 13
  28. 2026X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMscited by 13
  29. 2026Terminal-World: Scaling Terminal-Agent Environments via Agent Skillscited by 13
  30. 2025Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactionscited by 12
  31. 2024BianCang: A Traditional Chinese Medicine Large Language Modelcited by 11
  32. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbationcited by 11
  33. 2025Kvasir-VQA-x1: A Multimodal Dataset for Medical Reasoning and Robust MedVQA in Gastrointestinal Endoscopycited by 11
  34. 2025BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Datasetcited by 11
  35. 2026DocDancer: Towards Agentic Document-Grounded Information Seekingcited by 11
  36. 2026VideoVeritas: AI-Generated Video Detection via Perception Pretext Reinforcement Learningcited by 11
  37. 2026ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoningcited by 11
  38. 2025MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Userscited by 10

50 most cited of 345 All 345 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
  2. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  4. 2024Efficient Multimodal Large Language Models: A Survey
  5. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  6. 2023A Comprehensive Overview of Large Language Models
  7. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
  8. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints