# LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection

arXiv: https://arxiv.org/abs/2505.03793 · HTML: https://mlsanity.com/paper/2505.03793/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 57 references that are in the map.
- First posted on arXiv 1 May 2025.

## Builds on (most cited first, 30 of 47)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [A Convergence Theory for Deep Learning via Over-Parameterization](https://mlsanity.com/paper/1811.03962/) (2018)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Loss landscapes and optimization in over-parameterized non-linear systems and neural networks](https://mlsanity.com/paper/2003.00307/) (2020)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [A Bayesian Perspective on Generalization and Stochastic Gradient Descent](https://mlsanity.com/paper/1710.06451/) (2017)
- [Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel](https://mlsanity.com/paper/1810.05369/) (2018)

## Built on by (most cited first, 14 of 14)

- [JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation](https://mlsanity.com/paper/2509.22548/) (2025)
- [Plan and Budget: Effective and Efficient Test-Time Scaling on Reasoning Large Language Models](https://mlsanity.com/paper/2505.16122/) (2025)
- [LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing](https://mlsanity.com/paper/2507.00029/) (2025)
- [DynaWeb: Model-Based Reinforcement Learning of Web Agents](https://mlsanity.com/paper/2601.22149/) (2026)
- [Can Language Models Discover Scaling Laws?](https://mlsanity.com/paper/2507.21184/) (2025)
- [Train-before-Test Harmonizes Language Model Rankings](https://mlsanity.com/paper/2507.05195/) (2025)
- [Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning](https://mlsanity.com/paper/2602.17835/) (2026)
- [Prototype-Aligned Federated Soft-Prompts for Continual Web Personalization](https://mlsanity.com/paper/2603.27678/) (2026)
- [Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs](https://mlsanity.com/paper/2607.22039/) (2026)
- [Fine-Tuning Dynamics of In-Context Factual Recall in Transformers](https://mlsanity.com/paper/2605.27774/) (2026)
- [TuneAhead: Predicting Fine-tuning Performance Before Full Training Begins](https://mlsanity.com/paper/2606.17660/) (2026)
- [Beyond More Context: Retrieval Diversity Boosts Multi-Turn Intent Understanding](https://mlsanity.com/paper/2510.17940/) (2025)
- [A Risk Decomposition Framework for Pre-Hoc Fine-Tuning Prediction](https://mlsanity.com/paper/2606.17649/) (2026)
- [FrugalSOT - Frugal Search Over the Models](https://mlsanity.com/paper/2608.21621/) (2026)

## Abstract (arXiv)

The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks? In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a PAC-Bayesian Generalization Bound that unveils fine-tuning dynamics of LLMs and then introduce LENSLLM, a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed LENSLLM model and corresponding results at LensLLM.io.