# LQF: Linear Quadratic Fine-Tuning

arXiv: https://arxiv.org/abs/2012.11140 · HTML: https://mlsanity.com/paper/2012.11140/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 25 of 25)

- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [A Kernel-Based View of Language Model Fine-Tuning](https://mlsanity.com/paper/2210.05643/) (2022)
- [Mixed Differential Privacy in Computer Vision](https://mlsanity.com/paper/2203.11481/) (2022)
- [ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training](https://mlsanity.com/paper/2210.01738/) (2022)
- [The Journey, Not the Destination: How Data Guides Diffusion Models](https://mlsanity.com/paper/2312.06205/) (2023)
- [SAFE: Machine Unlearning With Shard Graphs](https://mlsanity.com/paper/2304.13169/) (2023)
- [TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels](https://mlsanity.com/paper/2207.06343/) (2022)
- [Diffusion Soup: Model Merging for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2406.08431/) (2024)
- [Tangent Model Composition for Ensembling and Continual Fine-tuning](https://mlsanity.com/paper/2307.08114/) (2023)
- [Taming AI Bots: Controllability of Neural States in Large Language Models](https://mlsanity.com/paper/2305.18449/) (2023)
- [SphereFed: Hyperspherical Federated Learning](https://mlsanity.com/paper/2207.09413/) (2022)
- [Training Data Protection with Compositional Diffusion Models](https://mlsanity.com/paper/2308.01937/) (2023)
- [Tangent Transformers for Composition, Privacy and Removal](https://mlsanity.com/paper/2307.08122/) (2023)
- [DLCFT: Deep Linear Continual Fine-Tuning for General Incremental Learning](https://mlsanity.com/paper/2208.08112/) (2022)
- [Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernel](https://mlsanity.com/paper/2209.15208/) (2022)
- [Train/Test-Time Adaptation with Retrieval](https://mlsanity.com/paper/2303.14333/) (2023)
- [DIVA: Dataset Derivative of a Learning Task](https://mlsanity.com/paper/2111.09785/) (2021)
- [Integral Continual Learning Along the Tangent Vector Field of Tasks](https://mlsanity.com/paper/2211.13108/) (2022)
- [Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature](https://mlsanity.com/paper/2602.17385/) (2026)
- [Towards Differential Relational Privacy and its use in Question Answering](https://mlsanity.com/paper/2203.16701/) (2022)
- [Thumb on the Scale: Optimal Loss Weighting in Last Layer Retraining](https://mlsanity.com/paper/2506.20025/) (2025)
- [Scene Uncertainty and the Wellington Posterior of Deterministic Image Classifiers](https://mlsanity.com/paper/2106.13870/) (2021)
- [Mathematics of Continual Learning](https://mlsanity.com/paper/2504.17963/) (2025)
- [Scalable Data Attribution via Forward-Only Test-Time Inference](https://mlsanity.com/paper/2511.19803/) (2025)

## Abstract (arXiv)

Classifiers that are linear in their parameters, and trained by optimizing a convex loss function, have predictable behavior with respect to changes in the training data, initial conditions, and optimization. Such desirable properties are absent in deep neural networks (DNNs), typically trained by non-linear fine-tuning of a pre-trained model. Previous attempts to linearize DNNs have led to interesting theoretical insights, but have not impacted the practice due to the substantial performance gap compared to standard non-linear optimization. We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning on most of real-world image classification tasks tested, thus enjoying the interpretability of linear models without incurring punishing losses in performance. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification: Leaky-ReLU instead of ReLU, mean squared loss instead of cross-entropy, and pre-conditioning using Kronecker factorization. None of these changes in isolation is sufficient to approach the performance of non-linear fine-tuning. When used in combination, they allow us to reach comparable performance, and even superior in the low-data regime, while enjoying the simplicity, robustness and interpretability of linear-quadratic optimization.