# Compressed Active Subspaces for Scalable Bayesian Inference

arXiv: https://arxiv.org/abs/2609.19539 · HTML: https://mlsanity.com/paper/2609.19539/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 17 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization](https://mlsanity.com/paper/1905.13727/) (2019)
- [Bayesian Low-rank Adaptation for Large Language Models](https://mlsanity.com/paper/2308.13111/) (2023)
- [Subspace Inference for Bayesian Deep Learning](https://mlsanity.com/paper/1907.07504/) (2019)
- [Learning Neural Network Subspaces](https://mlsanity.com/paper/2102.10472/) (2021)
- [Communication-efficient distributed SGD with Sketching](https://mlsanity.com/paper/1903.04488/) (2019)
- [Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence](https://mlsanity.com/paper/1505.02250/) (2015)
- [Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization](https://mlsanity.com/paper/2102.07245/) (2021)
- [Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details](https://mlsanity.com/paper/2108.11000/) (2021)
- [Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference](https://mlsanity.com/paper/2506.21408/) (2025)
- [Learning Active Subspaces for Effective and Scalable Uncertainty Quantification in Deep Neural Networks](https://mlsanity.com/paper/2309.03061/) (2023)
- [C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2505.17773/) (2025)
- [Training Bayesian Neural Networks with Sparse Subspace Variational Inference](https://mlsanity.com/paper/2402.11025/) (2024)
- [Uni-LoRA: One Vector is All You Need](https://mlsanity.com/paper/2506.00799/) (2025)
- [Spike-and-slab shrinkage priors for structurally sparse Bayesian neural networks](https://mlsanity.com/paper/2308.09104/) (2023)
- [Bayesian Semi-structured Subspace Inference](https://mlsanity.com/paper/2401.12950/) (2024)
- [Problem-dependent convergence bounds for randomized linear gradient compression](https://mlsanity.com/paper/2411.12898/) (2024)
- [Leveraging Active Subspaces to Capture Epistemic Model Uncertainty in Deep Generative Models for Molecular Design](https://mlsanity.com/paper/2405.00202/) (2024)
- [Enhancing Generative Molecular Design via Uncertainty-guided Fine-tuning of Variational Autoencoders](https://mlsanity.com/paper/2405.20573/) (2024)
- [Uncertainty-Aware Adaptation of Large Language Models for Protein-Protein Interaction Analysis](https://mlsanity.com/paper/2502.06173/) (2025)

## Abstract (arXiv)

Active subspace methods provide a framework for quantifying predictive uncertainty in high-dimensional models by identifying and performing inference along parameter directions that have the greatest influence on the model output. However, the construction of active subspaces requires storing many full-dimensional model gradients, which becomes prohibitive as model size increases. We address this limitation by proposing Compressed Active Subspaces (CAS), a scalable approach that first maps the model parameters to a compressed space using a structured isometric embedding and then constructs the active subspace within this reduced parameterization. Our approach substantially reduces the memory required for active subspace construction and enables Bayesian inference for large models where standard active subspace methods become impractical. We demonstrate the scalability of CAS on neural networks of increasing size while maintaining predictive performance and robust uncertainty estimates.