# RFFNet: Large-Scale Interpretable Kernel Methods via Random Fourier Features

arXiv: https://arxiv.org/abs/2211.06410 · HTML: https://mlsanity.com/paper/2211.06410/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 69 references that are in the map.
- First posted on arXiv 11 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Generalization Properties of Learning with Random Features](https://mlsanity.com/paper/1602.04474/) (2016)
- [On the Equivalence between Kernel Quadrature Rules and Random Feature Expansions](https://mlsanity.com/paper/1502.06800/) (2015)
- [FALKON: An Optimal Large Scale Kernel Method](https://mlsanity.com/paper/1705.10958/) (2017)
- [Revisiting the Nystrom Method for Improved Large-Scale Machine Learning](https://mlsanity.com/paper/1303.1849/) (2013)
- [Random Fourier Features for Kernel Ridge Regression: Approximation Bounds and Statistical Guarantees](https://mlsanity.com/paper/1804.09893/) (2018)
- [Diving into the shallows: a computational perspective on large-scale shallow learning](https://mlsanity.com/paper/1703.10622/) (2017)
- [Kernel Feature Selection via Conditional Covariance Minimization](https://mlsanity.com/paper/1707.01164/) (2017)
- [Testing Goodness of Fit of Conditional Density Models with Kernels](https://mlsanity.com/paper/2002.10271/) (2020)
- [A la Carte - Learning Fast Kernels](https://mlsanity.com/paper/1412.6493/) (2014)
- [Meta Two-Sample Testing: Learning Kernels for Testing with Limited Data](https://mlsanity.com/paper/2106.07636/) (2021)
- [Differentially Private Database Release via Kernel Mean Embeddings](https://mlsanity.com/paper/1710.01641/) (2017)
- [An Asymptotic Test for Conditional Independence using Analytic Kernel Embeddings](https://mlsanity.com/paper/2110.14868/) (2021)
- [Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes](https://mlsanity.com/paper/2111.04558/) (2021)
- [Random Gegenbauer Features for Scalable Kernel Methods](https://mlsanity.com/paper/2202.03474/) (2022)
- [On the Self-Penalization Phenomenon in Feature Selection](https://mlsanity.com/paper/2110.05852/) (2021)
- [Large-scale Nonlinear Variable Selection via Kernel Random Features](https://mlsanity.com/paper/1804.07169/) (2018)
- [Automated Spectral Kernel Learning](https://mlsanity.com/paper/1909.04894/) (2019)
- [Taming Nonconvexity in Kernel Feature Selection -- Favorable Properties of the Laplace Kernel](https://mlsanity.com/paper/2106.09387/) (2021)
- [Instrument Space Selection for Kernel Maximum Moment Restriction](https://mlsanity.com/paper/2106.03340/) (2021)
- [McKernel: A Library for Approximate Kernel Expansions in Log-linear Time](https://mlsanity.com/paper/1702.08159/) (2017)

## Abstract (arXiv)

Kernel methods provide a flexible and theoretically grounded approach to nonlinear and nonparametric learning. While memory and run-time requirements hinder their applicability to large datasets, many low-rank kernel approximations, such as random Fourier features, were recently developed to scale up such kernel methods. However, these scalable approaches are based on approximations of isotropic kernels, which cannot remove the influence of irrelevant features. In this work, we design random Fourier features for a family of automatic relevance determination (ARD) kernels, and introduce RFFNet, a new large-scale kernel method that learns the kernel relevances' on the fly via first-order stochastic optimization. We present an effective initialization scheme for the method's non-convex objective function, evaluate if hard-thresholding RFFNet's learned relevances yield a sensible rule for variable selection, and perform an extensive ablation study of RFFNet's components. Numerical validation on simulated and real-world data shows that our approach has a small memory footprint and run-time, achieves low prediction error, and effectively identifies relevant features, thus leading to more interpretable solutions. We supply users with an efficient, PyTorch-based library, that adheres to the scikit-learn standard API and code for fully reproducing our results.