# Improving Large Models with Small models: Lower Costs and Better Performance

arXiv: https://arxiv.org/abs/2406.15471 · HTML: https://mlsanity.com/paper/2406.15471/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Visual Prompt Multi-Modal Tracking](https://mlsanity.com/paper/2303.10826/) (2023)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)

## Built on by (most cited first, 14 of 14)

- [Logic Distillation: Learning from Code Function by Function for Decision-making Tasks](https://mlsanity.com/paper/2407.19405/) (2024)
- [TRIZ Agents: A Multi-Agent LLM Approach for TRIZ-Based Innovation](https://mlsanity.com/paper/2506.18783/) (2025)
- [DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correction](https://mlsanity.com/paper/2510.09211/) (2025)
- [KKA: Improving Vision Anomaly Detection through Anomaly-related Knowledge from Large Language Models](https://mlsanity.com/paper/2502.14880/) (2025)
- [An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation](https://mlsanity.com/paper/2505.20182/) (2025)
- [Structuring Radiology Reports: Challenging LLMs with Lightweight Models](https://mlsanity.com/paper/2506.00200/) (2025)
- [Easy Adaptation: An Efficient Task-Specific Knowledge Injection Method for Large Models in Resource-Constrained Environments](https://mlsanity.com/paper/2512.17771/) (2025)
- [LeForecast: Enterprise Hybrid Forecast by Time Series Intelligence](https://mlsanity.com/paper/2503.22747/) (2025)
- [FAF: A Feature-Adaptive Framework for Few-Shot Time Series Forecasting](https://mlsanity.com/paper/2506.19567/) (2025)
- [Semantic Encryption: Secure and Effective Interaction with Cloud-based Large Language Models via Semantic Transformation](https://mlsanity.com/paper/2508.01638/) (2025)
- [MAFIG: Multi-agent Driven Formal Instruction Generation Framework](https://mlsanity.com/paper/2604.10989/) (2026)
- [What Will Happen Next: Large Models-Driven Deduction for Emergency Instances](https://mlsanity.com/paper/2605.08599/) (2026)
- [IFPV: An Integrated Multi-Agent Framework for Generative Operational Planning and High-Fidelity Plan Verification](https://mlsanity.com/paper/2605.14851/) (2026)
- [When Efficiency Backfires: Cascading LLMs Trigger Cascade Failure under Adversarial Attack](https://mlsanity.com/paper/2605.17288/) (2026)

## Abstract (arXiv)

Pretrained large models (PLMs), such as ChatGPT, have demonstrated remarkable performance across diverse tasks. However, the significant computational requirements of PLMs have discouraged most product teams from running or fine-tuning them. In such cases, to harness the exceptional performance of PLMs, one must rely on expensive APIs, thereby exacerbating the economic burden. Despite the overall inferior performance of small models, in specific distributions, they can achieve comparable or even superior results. Consequently, some input can be processed exclusively by small models. On the other hand, certain tasks can be broken down into multiple subtasks, some of which can be completed without powerful capabilities. Under these circumstances, small models can handle the simple subtasks, allowing large models to focus on challenging subtasks, thus improving the performance. We propose Data Shunt^+ (DS^+), a general paradigm for collaboration of small and large models. DS^+ not only substantially reduces the cost associated with querying large models but also effectively improves large models' performance. For instance, ChatGPT achieves an accuracy of 94.43% on Amazon Product sentiment analysis, and DS^+ achieves an accuracy of 95.64%, while the cost has been reduced to only 31.18%. Besides, experiments also prove that the proposed collaborative-based paradigm can better inject specific task knowledge into PLMs compared to fine-tuning.