# Ghost-Connect Net: A Generalization-Enhanced Guidance For Sparse Deep Networks Under Distribution Shifts

arXiv: https://arxiv.org/abs/2411.09199 · HTML: https://mlsanity.com/paper/2411.09199/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Pruning Neural Networks at Initialization: Why are We Missing the Mark?](https://mlsanity.com/paper/2009.08576/) (2020)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [Layer-adaptive sparsity for the Magnitude-based Pruning](https://mlsanity.com/paper/2010.07611/) (2020)
- [A Unified View of Label Shift Estimation](https://mlsanity.com/paper/2003.07554/) (2020)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [Neural Machine Translation: A Review and Survey](https://mlsanity.com/paper/1912.02047/) (2019)
- [Neural Pruning via Growing Regularization](https://mlsanity.com/paper/2012.09243/) (2020)
- [EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning](https://mlsanity.com/paper/2007.02491/) (2020)
- [A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness](https://mlsanity.com/paper/2106.09129/) (2021)
- [The Elastic Lottery Ticket Hypothesis](https://mlsanity.com/paper/2103.16547/) (2021)
- [MIT Advanced Vehicle Technology Study: Large-Scale Naturalistic Driving Study of Driver Behavior and Interaction with Automation](https://mlsanity.com/paper/1711.06976/) (2017)
- [DeepPicar: A Low-cost Deep Neural Network-based Autonomous Car](https://mlsanity.com/paper/1712.08644/) (2017)
- [Theoretical Understanding of the Information Flow on Continual Learning Performance](https://mlsanity.com/paper/2204.12010/) (2022)

## Abstract (arXiv)

Sparse deep neural networks (DNNs) excel in real-world applications like robotics and computer vision, by reducing computational demands that hinder usability. However, recent studies aim to boost DNN efficiency by trimming redundant neurons or filters based on task relevance, but neglect their adaptability to distribution shifts. We aim to enhance these existing techniques by introducing a companion network, Ghost Connect-Net (GC-Net), to monitor the connections in the original network with distribution generalization advantage. GC-Net's weights represent connectivity measurements between consecutive layers of the original network. After pruning GC-Net, the pruned locations are mapped back to the original network as pruned connections, allowing for the combination of magnitude and connectivity-based pruning methods. Experimental results using common DNN benchmarks, such as CIFAR-10, Fashion MNIST, and Tiny ImageNet show promising results for hybridizing the method, and using GC-Net guidance for later layers of a network and direct pruning on earlier layers. We provide theoretical foundations for GC-Net's approach to improving generalization under distribution shifts.