# Tuning Language Models by Proxy

arXiv: https://arxiv.org/abs/2401.08565 · HTML: https://mlsanity.com/paper/2401.08565/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 48 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Builds on (most cited first, 30 of 47)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts](https://mlsanity.com/paper/2105.03023/) (2021)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [GeDi: Generative Discriminator Guided Sequence Generation](https://mlsanity.com/paper/2009.06367/) (2020)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)

## Built on by (most cited first, 30 of 89)

- [Weak-to-Strong Jailbreaking on Large Language Models](https://mlsanity.com/paper/2401.17256/) (2024)
- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference](https://mlsanity.com/paper/2406.08607/) (2024)
- [Decoding-time Realignment of Language Models](https://mlsanity.com/paper/2402.02992/) (2024)
- [Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimization](https://mlsanity.com/paper/2404.11531/) (2024)
- [Learning or Self-aligning? Rethinking Instruction Fine-tuning](https://mlsanity.com/paper/2402.18243/) (2024)
- [Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B People](https://mlsanity.com/paper/2403.03640/) (2024)
- [Offset Unlearning for Large Language Models](https://mlsanity.com/paper/2404.11045/) (2024)
- [MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoning](https://mlsanity.com/paper/2405.03000/) (2024)
- [From Distributional to Overton Pluralism: Investigating Large Language Model Alignment](https://mlsanity.com/paper/2406.17692/) (2024)
- [Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models](https://mlsanity.com/paper/2405.19262/) (2024)
- [Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMs](https://mlsanity.com/paper/2502.19148/) (2025)
- [Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence](https://mlsanity.com/paper/2410.11163/) (2024)
- [A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models](https://mlsanity.com/paper/2504.07070/) (2025)
- [Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems](https://mlsanity.com/paper/2502.04510/) (2025)
- [Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model](https://mlsanity.com/paper/2410.18640/) (2024)
- [Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment](https://mlsanity.com/paper/2405.17871/) (2024)
- [Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast](https://mlsanity.com/paper/2405.14507/) (2024)
- [Inference-Time Language Model Alignment via Integrated Value Guidance](https://mlsanity.com/paper/2409.17819/) (2024)
- [A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations](https://mlsanity.com/paper/2406.10303/) (2024)
- [Learning From Correctness Without Prompting Makes LLM Efficient Reasoner](https://mlsanity.com/paper/2403.19094/) (2024)
- [MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usability](https://mlsanity.com/paper/2405.14488/) (2024)
- [Qsharp: Provably Optimal Distributional RL for LLM Post-Training](https://mlsanity.com/paper/2502.20548/) (2025)
- [Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding](https://mlsanity.com/paper/2406.12295/) (2024)
- [CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steering](https://mlsanity.com/paper/2507.04756/) (2025)
- [Shadow-FT: Tuning Instruct Model via Training on Paired Base Model](https://mlsanity.com/paper/2505.12716/) (2025)
- [Don't Throw Away Your Pretrained Model](https://mlsanity.com/paper/2510.09913/) (2025)
- [On Giant's Shoulders: Effortless Weak to Strong by Dynamic Logits Fusion](https://mlsanity.com/paper/2406.15480/) (2024)
- [Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learning](https://mlsanity.com/paper/2410.10074/) (2024)
- [Robust Multi-Objective Controlled Decoding of Large Language Models](https://mlsanity.com/paper/2503.08796/) (2025)

## Abstract (arXiv)

Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.