# Supervised Dictionary Learning with Auxiliary Covariates

arXiv: https://arxiv.org/abs/2206.06774 · HTML: https://mlsanity.com/paper/2206.06774/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 78 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 19 of 19)

- [Coordinate Descent Algorithms](https://mlsanity.com/paper/1502.04759/) (2015)
- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [Low-rank Solutions of Linear Matrix Equations via Procrustes Flow](https://mlsanity.com/paper/1507.03566/) (2015)
- [Latent Dirichlet Allocation (LDA) and Topic modeling: models, applications, a survey](https://mlsanity.com/paper/1711.04305/) (2017)
- [Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descent](https://mlsanity.com/paper/2005.08898/) (2020)
- [On Markov Chain Gradient Descent](https://mlsanity.com/paper/1809.04216/) (2018)
- [Non-square matrix sensing without spurious local minima via the Burer-Monteiro approach](https://mlsanity.com/paper/1609.03240/) (2016)
- [A Convergent Gradient Descent Algorithm for Rank Minimization and Semidefinite Programming from Random Linear Measurements](https://mlsanity.com/paper/1506.06081/) (2015)
- [Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization](https://mlsanity.com/paper/1306.4650/) (2013)
- [Finding Low-Rank Solutions via Non-Convex Matrix Factorization, Efficiently and Provably](https://mlsanity.com/paper/1606.03168/) (2016)
- [A Unified Computational and Statistical Framework for Nonconvex Low-Rank Matrix Estimation](https://mlsanity.com/paper/1610.05275/) (2016)
- [Convergence guarantees for a class of non-convex and non-smooth optimization problems](https://mlsanity.com/paper/1804.09629/) (2018)
- [UCB-based Algorithms for Multinomial Logistic Regression Bandits](https://mlsanity.com/paper/2103.11489/) (2021)
- [Non-negative Matrix Factorization: Robust Extraction of Extended Structures](https://mlsanity.com/paper/1712.10317/) (2017)
- [Block majorization-minimization with diminishing radius for constrained nonsmooth nonconvex optimization](https://mlsanity.com/paper/2012.03503/) (2020)
- [On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization](https://mlsanity.com/paper/1901.10682/) (2019)
- [Semi-supervised NMF Models for Topic Modeling in Learning Tasks](https://mlsanity.com/paper/2010.07956/) (2020)
- [Supervised Dictionary Learning and Sparse Representation-A Review](https://mlsanity.com/paper/1502.05928/) (2015)
- [Structure-Aware Classification using Supervised Dictionary Learning](https://mlsanity.com/paper/1609.09199/) (2016)

## Built on by (most cited first, 1 of 1)

- [A latent linear model for nonlinear coupled oscillators on graphs](https://mlsanity.com/paper/2311.14910/) (2023)

## Abstract (arXiv)

Supervised dictionary learning (SDL) is a classical machine learning method that simultaneously seeks feature extraction and classification tasks, which are not necessarily a priori aligned objectives. The goal of SDL is to learn a class-discriminative dictionary, which is a set of latent feature vectors that can well-explain both the features as well as labels of observed data. In this paper, we provide a systematic study of SDL, including the theory, algorithm, and applications of SDL. First, we provide a novel framework that `lifts' SDL as a convex problem in a combined factor space and propose a low-rank projected gradient descent algorithm that converges exponentially to the global minimizer of the objective. We also formulate generative models of SDL and provide global estimation guarantees of the true parameters depending on the hyperparameter regime. Second, viewed as a nonconvex constrained optimization problem, we provided an efficient block coordinate descent algorithm for SDL that is guaranteed to find an varepsilon-stationary point of the objective in O(varepsilon-1(log varepsilon-1)2) iterations. For the corresponding generative model, we establish a novel non-asymptotic local consistency result for constrained and regularized maximum likelihood estimation problems, which may be of independent interest. Third, we apply SDL for imbalanced document classification by supervised topic modeling and also for pneumonia detection from chest X-ray images. We also provide simulation studies to demonstrate that SDL becomes more effective when there is a discrepancy between the best reconstructive and the best discriminative dictionaries.