mlsanityData to 14 Oct 2026

CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Models

arXiv 2305.16876 ↗First posted 23 May 2023cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

Methods for adapting language models (LMs) to new tasks and domains have traditionally assumed white-box access to the model, and work by modifying its parameters. However, this is incompatible with a recent trend in the field, where the highest quality models are only available as black-boxes through inference APIs. Even when the model weights are available, the computational cost of fine-tuning large LMs can be prohibitive for most practitioners. In this work, we present a lightweight method for adapting large LMs to new domains and tasks, assuming no access to their weights or intermediate activations. Our approach fine-tunes a small white-box LM and combines it with the large black-box LM at the probability level through a small network, learned on a small validation set. We validate our approach by adapting a large LM (OPT-30B) to several domains and a downstream task (machine translation), observing improved performance in all cases, of up to 9%, while using a domain expert 23x smaller.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

26 of 38 references.

References · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
14 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  2. 2015Image-based Recommendations on Styles and Substitutescited by 706
  3. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 520
  4. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  5. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  6. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  7. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  8. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  9. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  10. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 75
  11. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 60
  12. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  13. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  14. 2023kNN-Adapter: Efficient Domain Adaptation for Black-Box Language Modelscited by 14

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 11; 2025: 12.

26 papers.

Sort:

By area:

  1. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  2. 2024Tuning Language Models by Proxycited by 89
  3. 2024Learning to Decode Collaboratively with Multiple Language Modelscited by 62
  4. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
  5. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  6. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  7. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 41
  8. 2024Offset Unlearning for Large Language Modelscited by 36
  9. 2024CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Followingcited by 33
  10. 2023Measuring the Robustness of NLP Models to Domain Shiftscited by 11
  11. 2024BBox-Adapter: Lightweight Adapting for Black-Box Large Language Modelscited by 11
  12. 2024PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patchescited by 7
14 more
  1. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  2. 2023On-the-Fly Fusion of Large Language Models and Machine Translationcited by 5
  3. 2024FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devicescited by 5
  4. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  5. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 4
  6. 2023Controlled Text Generation for Black-box Language Models via Score-based Progressive Editorcited by 2
  7. 2025Large-Small Model Collaboration for Enhancing Edge-Deployed Small Modelscited by 2
  8. 2025Logit Arithmetic Elicits Long Reasoning Capabilities Without Trainingcited by 2
  9. 2025DICE: Structured Reasoning in LLMs through SLM-Guided Chain-of-Thought Correctioncited by 2
  10. 2025Delta Sampling: Data-Free Knowledge Transfer Across Diffusion Modelscited by 2
  11. 2025Generalized and Personalized Federated Learning with Black-Box Foundation Models via Orthogonal Transformationscited by 1
  12. 2025Daunce: Data Attribution through Uncertainty Estimationcited by 1
  13. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  14. 2025Advanced Black-Box Tuning of Large Language Models with Limited API Callscited by 1