mlsanityData to 14 Oct 2026

Interpolating Compressed Parameter Subspaces

arXiv 2205.09891 ↗First posted 19 May 2022cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Inspired by recent work on neural subspaces and mode connectivity, we revisit parameter subspace sampling for shifted and/or interpolatable input distributions (instead of a single, unshifted distribution). We enforce a compressed geometric structure upon a set of trained parameters mapped to a set of train-time distributions, denoting the resulting subspaces as Compressed Parameter Subspaces (CPS). We show the success and failure modes of the types of shifted distributions whose optimal parameters reside in the CPS. We find that ensembling point-estimates within a CPS can yield a high average accuracy across a range of test-time distributions, including backdoor, adversarial, permutation, stylization and rotation perturbations. We also find that the CPS can contain low-loss point-estimates for various task shifts (albeit interpolated, perturbed, unseen or non-identical coarse labels). We further demonstrate this property in a continual learning setting with CIFAR100.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 57 references that are in the map. First posted on arXiv 19 May 2022.

Builds on

41 of 57 references.

References · 41

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  9. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  10. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  11. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,082
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
29 more
  1. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  3. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  4. 2015Return of Frustratingly Easy Domain Adaptationcited by 391
  5. 2020What is being transferred in transfer learning?cited by 368
  6. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  7. 2015Scalable Bayesian Optimization Using Deep Neural Networkscited by 284
  8. 2019Continual learning with hypernetworkscited by 273
  9. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  10. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  11. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 158
  12. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  13. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 145
  14. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  15. 2020Hybrid Models for Open Set Recognitioncited by 109
  16. 2019Cross-Domain Transferability of Adversarial Perturbationscited by 92
  17. 2020Strong Data Augmentation Sanitizes Poisoning and Backdoor Attacks Without an Accuracy Tradeoffcited by 89
  18. 2018Unsupervised Learning via Meta-Learningcited by 82
  19. 2021Learning Neural Network Subspacescited by 82
  20. 2018Unsupervised Meta-Learning For Few-Shot Image Classificationcited by 75
  21. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  22. 2019Superposition of many models into onecited by 64
  23. 2021Meta-Learning with Fewer Tasks through Task Interpolationcited by 45
  24. 2021On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learningcited by 29
  25. 2021Self-Adversarial Disentangling for Specific Domain Adaptationcited by 29
  26. 2020A Data Augmentation-based Defense Method Against Adversarial Attacks in Neural Networkscited by 7
  27. 2021A Synergetic Attack against Neural Network Classifiers combining Backdoor and Adversarial Examplescited by 6
  28. 2022Learn2Weight: Parameter Adaptation against Similar-domain Adversarial Attackscited by 5
  29. 2022Backdoors Stuck At The Frontdoor: Multi-Agent Backdoor Attacks That Backfirecited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1.

3 papers.

  1. 2022Multiple Modes for Continual Learningcited by 2
  2. 2022Cross-Reality Re-Rendering: Manipulating between Digital and Physical Realitiescited by 0
  3. 2023Projected Subnetworks Scale Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time
  2. 2026Learning Functional Subspaces for Neural Network Compression
  3. 2024How Do Flow Matching Models Memorize and Generalize in Sample Data Subspaces?
  4. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Space
  5. 2025High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation
  6. 2026Compressed Computation is (probably) not Computation in Superposition
  7. 2026Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning
  8. 2026MetaSAEs: Joint Training with a Decomposability Penalty Produces More Atomic Sparse Autoencoder Latents