mlsanityData to 14 Oct 2026

On consequences of finetuning on data with highly discriminative features

arXiv 2310.19537 ↗First posted 30 Oct 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In the era of transfer learning, training neural networks from scratch is becoming obsolete. Transfer learning leverages prior knowledge for new tasks, conserving computational resources. While its advantages are well-documented, we uncover a notable drawback: networks tend to prioritize basic data patterns, forsaking valuable pre-learned features. We term this behavior "feature erosion" and analyze its impact on network performance and internal representations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 18 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

15 of 18 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  3. 2014How transferable are features in deep neural networks?cited by 1,225
  4. 2019Machine Unlearningcited by 1,061
  5. 2017A Closer Look at Memorization in Deep Networkscited by 942
  6. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  7. 2018Rethinking the Value of Network Pruningcited by 599
  8. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  9. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 342
  10. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 267
  11. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
3 more
  1. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 161
  2. 2019Fairness in Deep Learning: A Computational Perspectivecited by 123
  3. 2023Is forgetting less a good inductive bias for forward transfer?cited by 18

Built on by

Not cited yet.