mlsanityData to 14 Oct 2026

Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuning

arXiv 2604.16332 ↗First posted 12 Mar 2026cs.LG cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We find that LoRA fine-tuning exhibits un-learning on contested examples: items with high annotator disagreement show increasing loss during training, a qualitatively distinct pattern largely absent under full fine-tuning and consistent across all six models tested (four encoder, two decoder-only). This discovery emerges from correlating annotation entropy, computed from ChaosNLI's 100 labels per example, with per-example area under the loss curve (AULC) on SNLI and MNLI. The correlation is positive in all 25 conditions tested (Spearman ρ = 0.06-0.43), with decoder-only models showing stronger correlations than encoders at matched LoRA rank. The effect survives partial-correlation controls and replicates across seeds and datasets. A preliminary noise-injection experiment is consistent with these findings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 42 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

28 of 42 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  7. 2017A Closer Look at Memorization in Deep Networkscited by 942
  8. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  11. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  12. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
16 more
  1. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  2. 2024LoRA Learns Less and Forgets Lesscited by 328
  3. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  4. 2020Identifying Mislabeled Data using the Area Under the Margin Rankingcited by 261
  5. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  6. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  7. 2019Let's Agree to Agree: Neural Networks Share Classification Order on Real Datasetscited by 56
  8. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
  9. 2021Embracing Ambiguity: Shifting the Training Target of NLI Modelscited by 23
  10. 2025EvoLM: In Search of Lost Language Model Training Dynamicscited by 16
  11. 2026Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspectivecited by 6
  12. 2025Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learningcited by 5
  13. 2025CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamicscited by 4
  14. 2026Beware of the Batch Size: Hyperparameter Bias in Evaluating LoRAcited by 4
  15. 2025Alignment Data Map for Efficient Preference Data Selection and Diagnosiscited by 2
  16. 2025When is dataset cartography ineffective? Using training dynamics does not improve robustness against Adversarial SQuADcited by 1

Built on by

Not cited yet.