# Growing a Brain: Fine-Tuning by Increasing Model Capacity

arXiv: https://arxiv.org/abs/1907.07844 · HTML: https://mlsanity.com/paper/1907.07844/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2019.

## Built on by (most cited first, 30 of 43)

- [Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations](https://mlsanity.com/paper/2103.06342/) (2021)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [RECALL: Replay-based Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2108.03673/) (2021)
- [Continual Backprop: Stochastic Gradient Descent with Persistent Randomness](https://mlsanity.com/paper/2108.06325/) (2021)
- [Computationally Budgeted Continual Learning: What Does Matter?](https://mlsanity.com/paper/2303.11165/) (2023)
- [Class-Incremental Learning with Strong Pre-trained Models](https://mlsanity.com/paper/2204.03634/) (2022)
- [A Review of Deep Transfer Learning and Recent Advancements](https://mlsanity.com/paper/2201.09679/) (2022)
- [Self-Supervised Training Enhances Online Continual Learning](https://mlsanity.com/paper/2103.14010/) (2021)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [Language Models as Black-Box Optimizers for Vision-Language Models](https://mlsanity.com/paper/2309.05950/) (2023)
- [Revisiting Few-Shot Object Detection with Vision-Language Models](https://mlsanity.com/paper/2312.14494/) (2023)
- [Continual Segment: Towards a Single, Unified and Accessible Continual Segmentation Model of 143 Whole-body Organs in CT Scans](https://mlsanity.com/paper/2302.00162/) (2023)
- [On Anytime Learning at Macroscale](https://mlsanity.com/paper/2106.09563/) (2021)
- [Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models](https://mlsanity.com/paper/2505.20612/) (2025)
- [Fine-Tuning Games: Bargaining and Adaptation for General-Purpose Models](https://mlsanity.com/paper/2308.04399/) (2023)
- [Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale](https://mlsanity.com/paper/2510.24963/) (2025)
- [Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation](https://mlsanity.com/paper/2404.13425/) (2024)
- [An Analysis of Initial Training Strategies for Exemplar-Free Class-Incremental Learning](https://mlsanity.com/paper/2308.11677/) (2023)
- [Modality-Incremental Learning with Disjoint Relevance Mapping Networks for Image-based Semantic Segmentation](https://mlsanity.com/paper/2411.17610/) (2024)
- [Rethinking Task-Incremental Learning Baselines](https://mlsanity.com/paper/2205.11367/) (2022)
- [Queried Unlabeled Data Improves and Robustifies Class-Incremental Learning](https://mlsanity.com/paper/2206.07842/) (2022)
- [Distilling BlackBox to Interpretable models for Efficient Transfer Learning](https://mlsanity.com/paper/2305.17303/) (2023)
- [RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation](https://mlsanity.com/paper/2309.10479/) (2023)
- [Modular Adaptation for Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2104.00619/) (2021)
- [Cold Start Streaming Learning for Deep Networks](https://mlsanity.com/paper/2211.04624/) (2022)
- [Lifelong Learning and Selective Forgetting via Contrastive Strategy](https://mlsanity.com/paper/2405.18663/) (2024)
- [SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Framework](https://mlsanity.com/paper/2411.19234/) (2024)
- [Neural Supervised Domain Adaptation by Augmenting Pre-trained Models with Random Units](https://mlsanity.com/paper/2106.04935/) (2021)
- [A Comparative Study of Calibration Methods for Imbalanced Class Incremental Learning](https://mlsanity.com/paper/2202.00386/) (2022)
- [APP: Anytime Progressive Pruning](https://mlsanity.com/paper/2204.01640/) (2022)

## Abstract (arXiv)

CNNs have made an undeniable impact on computer vision through the ability to learn high-capacity models with large annotated training sets. One of their remarkable properties is the ability to transfer knowledge from a large source dataset to a (typically smaller) target dataset. This is usually accomplished through fine-tuning a fixed-size network on new target data. Indeed, virtually every contemporary visual recognition system makes use of fine-tuning to transfer knowledge from ImageNet. In this work, we analyze what components and parameters change during fine-tuning, and discover that increasing model capacity allows for more natural model adaptation through fine-tuning. By making an analogy to developmental learning, we demonstrate that "growing" a CNN with additional units, either by widening existing layers or deepening the overall network, significantly outperforms classic fine-tuning approaches. But in order to properly grow a network, we show that newly-added units must be appropriately normalized to allow for a pace of learning that is consistent with existing units. We empirically validate our approach on several benchmark datasets, producing state-of-the-art results.