mlsanityData to 7 Oct 2026

Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks

arXiv 2505.08022 ↗First posted 12 May 2025cs.LG cs.NA math.NAcited by 5

Not on the map. Why?

Abstract (arXiv)

Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing accuracy on clean data. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94% compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

37 of 55 references.

References · 37

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
  6. 2016Adversarial Machine Learning at Scalecited by 934
  7. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  8. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  10. 2019Nesterov Accelerated Gradient and Scale Invariance for Adversarial Attackscited by 352
  11. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  12. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
25 more
  1. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  2. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  3. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  4. 2018Sorting out Lipschitz function approximationcited by 211
  5. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  6. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  7. 2020Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transformcited by 79
  8. 2018Adv-BNN: Improved Adversarial Defense through Robust Bayesian Neural Networkcited by 68
  9. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  10. 2017All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulationcited by 58
  11. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  12. 2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
  13. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  14. 2024DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distributioncited by 41
  15. 2021Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attackscited by 33
  16. 2022Universal Adversarial Examples in Remote Sensing: Methodology and Benchmarkcited by 24
  17. 2021On the Effect of Pruning on Adversarial Robustnesscited by 23
  18. 2017Generative Adversarial Trainer: Defense to Adversarial Perturbations with GANcited by 20
  19. 2018Adversarial Examples in Remote Sensingcited by 19
  20. 2023Robust low-rank training via approximate orthonormal constraintscited by 12
  21. 2022Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustnesscited by 10
  22. 2024Federated Dynamical Low-Rank Training with Global Loss Convergence Guaranteescited by 7
  23. 2024(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Numbercited by 5
  24. 2024GeoLoRA: Geometric integration for parameter efficient fine-tuningcited by 5
  25. 2025An Augmented Backward-Corrected Projector Splitting Integrator for Dynamical Low-Rank Trainingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  2. 2025A geometric framework for momentum-based optimizers for low-rank trainingcited by 5
  3. 2025Mitigating Subject Dependency in EEG Decoding with Subject-Specific Low-Rank Adapterscited by 4
  4. 2026Compressing Vision Transformers in Geospatial Transfer Learning with Manifold-Constrained Optimizationcited by 0
  5. 2026RepUCB: Representation Learning-Based UCB for Heterogeneous Multi-Task Linear Banditscited by 0