# Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning

arXiv: https://arxiv.org/abs/2604.16332 · HTML: https://mlsanity.com/paper/2604.16332/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 42 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Identifying Mislabeled Data using the Area Under the Margin Ranking](https://mlsanity.com/paper/2001.10528/) (2020)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasets](https://mlsanity.com/paper/1905.10854/) (2019)
- [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023)
- [Embracing Ambiguity: Shifting the Training Target of NLI Models](https://mlsanity.com/paper/2106.03020/) (2021)
- [EvoLM: In Search of Lost Language Model Training Dynamics](https://mlsanity.com/paper/2506.16029/) (2025)
- [Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective](https://mlsanity.com/paper/2601.03154/) (2026)
- [Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning](https://mlsanity.com/paper/2506.05447/) (2025)
- [CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics](https://mlsanity.com/paper/2507.03004/) (2025)
- [Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRA](https://mlsanity.com/paper/2602.09492/) (2026)
- [Alignment Data Map for Efficient Preference Data Selection and Diagnosis](https://mlsanity.com/paper/2505.23114/) (2025)
- [When is dataset cartography ineffective? Using training dynamics does not improve robustness against Adversarial SQuAD](https://mlsanity.com/paper/2503.18290/) (2025)

## Abstract (arXiv)

We find that LoRA fine-tuning exhibits un-learning on contested examples: items with high annotator disagreement show increasing loss during training, a qualitatively distinct pattern largely absent under full fine-tuning and consistent across all six models tested (four encoder, two decoder-only). This discovery emerges from correlating annotation entropy, computed from ChaosNLI's 100 labels per example, with per-example area under the loss curve (AULC) on SNLI and MNLI. The correlation is positive in all 25 conditions tested (Spearman ρ = 0.06-0.43), with decoder-only models showing stronger correlations than encoders at matched LoRA rank. The effect survives partial-correlation controls and replicates across seeds and datasets. A preliminary noise-injection experiment is consistent with these findings.