# Reducing the Adaptation Gap Through Reachable Fisher Geometry

arXiv: https://arxiv.org/abs/2609.36329 · HTML: https://mlsanity.com/paper/2609.36329/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 96 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [DoRA: Weight-Decomposed Low-Rank Adaptation](https://mlsanity.com/paper/2402.09353/) (2024)
- [A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA](https://mlsanity.com/paper/2312.03732/) (2023)
- [BackPACK: Packing more into backprop](https://mlsanity.com/paper/1912.10985/) (2019)
- [MEDFAIR: Benchmarking Fairness for Medical Imaging](https://mlsanity.com/paper/2210.01725/) (2022)
- [Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunity](https://mlsanity.com/paper/2305.08252/) (2023)
- [Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape](https://mlsanity.com/paper/2409.14396/) (2024)
- [On Fairness of Low-Rank Adaptation of Large Models](https://mlsanity.com/paper/2405.17512/) (2024)
- [FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation](https://mlsanity.com/paper/2410.17358/) (2024)
- [FISH-Tuning: Enhancing PEFT Methods with Fisher Information](https://mlsanity.com/paper/2504.04050/) (2025)
- [Subgroups Matter for Robust Bias Mitigation](https://mlsanity.com/paper/2505.21363/) (2025)
- [Learning in the Fisher Subspace: A Guided Initialization for LoRA Fine-Tuning](https://mlsanity.com/paper/2605.01046/) (2026)
- [FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization](https://mlsanity.com/paper/2503.22934/) (2025)
- [Ethical Fairness in Ubiquitous Health Sensing without Known Attributes](https://mlsanity.com/paper/2603.13373/) (2026)
- [FairLLaVA: Fairness-Aware Parameter-Efficient Fine-Tuning for Large Vision-Language Assistants](https://mlsanity.com/paper/2603.26008/) (2026)
- [FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2605.29317/) (2026)

## Abstract (arXiv)

Parameter-efficient fine-tuning (PEFT) determines not only how many parameters are trained, but also which local directions a model can move in, so similar adapters can affect subgroup losses differently. Since curvature matrices are infeasible to form at adapter scale, scalar summaries such as the Fisher trace are often used instead. We study what the trace reveals and what it loses through the reachable Fisher: each subgroup's full-model Fisher pulled back through the adapter Jacobian. Under likelihood losses, it represents the Gauss-Newton curvature accessible to the adapter, and its trace can be computed from score-gradient norms without forming the full matrix. Under matched subgroup gradients, a positive-definite reachable-Fisher difference, with a margin exceeding the Hessian-Fisher defect, implies that every sufficiently small nonzero model-changing update increases the signed gap. In contrast, the restricted operator norm determines worst-case quadratic change, while a matrix-free Frobenius discrepancy bounds its reachable-Fisher component. Trace alone cannot certify definiteness or control matrix mismatch. Equal traces rule out a positive-definite difference but can still hide large operator discrepancies. Across 306 single-seed models, higher trace accompanies greater subgroup difficulty in 75.7 percent of 1,218 eligible evaluations, while trace matching reduces the best-worst subgroup gap in all 30 dataset-encoder-adapter combinations. However, held-out audits show that operator discrepancy decreases in 23 of 30 combinations, while the unbiased squared-Frobenius statistic decreases in only 16 of 30. Fisher trace is therefore a scalable diagnostic and training heuristic, but not a certificate of local gap behavior or matrix alignment.