# UpgradeBench: A Decision-Centric Benchmark for Upgrading Fine-Tuned LLM Specialists

arXiv: https://arxiv.org/abs/2608.20918 · HTML: https://mlsanity.com/paper/2608.20918/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 43 of its 46 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 30 of 43)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://mlsanity.com/paper/2311.03099/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [AI Agents That Matter](https://mlsanity.com/paper/2407.01502/) (2024)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://mlsanity.com/paper/2311.03285/) (2023)
- [BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices](https://mlsanity.com/paper/2411.12990/) (2024)
- [Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification](https://mlsanity.com/paper/2406.08660/) (2024)
- [Positive-Congruent Training: Towards Regression-Free Model Updates](https://mlsanity.com/paper/2011.09161/) (2020)
- [Chat Vector: A Simple Approach to Equip LLMs with Instruction Following and Model Alignment in New Languages](https://mlsanity.com/paper/2310.04799/) (2023)
- [Densing Law of LLMs](https://mlsanity.com/paper/2412.04315/) (2024)

## Abstract (arXiv)

Organizations maintain task-specific adapters for open-weight language models, and each new base-model release forces a migration decision: retain existing specialists, port adapters, refresh from preserved behavior, or retrain. Prior transfer work evaluates isolated model pairs, without studying these choices across real model release sequences. We present UpgradeBench, a decision-driven longitudinal benchmark covering four consecutive Qwen releases, one continuation checkpoint, six tasks, and two model sizes, augmented by OLMo checkpoints with known training lineage. The benchmark disentangles three core questions: whether a new checkpoint improves fixed-recipe retrained specialist performance, whether specialization assets transfer across versions, and what recovery resources are usable. We observe upgrade gains differ across task-scale-release episodes: some retrained baselines improve while others stay within training noise, with durability ranging from under one release interval for text-to-SQL to over fourteen months for intent classification. Direct adapter copying depends neither on architecture nor model family: on OLMo, retention drops from 0.88-0.99 at 46B-token continued pretraining to zero at 2.9T tokens; annealing and model souping introduce no extra harm, with portability decaying with continued-pretraining distance. Given preserved input data, teacher relabeling recovers target-base specialists without fresh gold annotations, though compute savings are not guaranteed. Simulating a fixed decision policy over 33 upgrade episodes yields 0.37pp mean quality regret with zero behavioral regressions at one-third the compute and label cost of full retraining. A lightweight CKA probe over 256 prompts predicts cross-version adapter portability (Spearman 0.74 across eight model pairs). We release per-example predictions, cost logs, split manifests, and evaluation code.