mlsanityData to 7 Oct 2026

Toward Large Kernel Models

arXiv 2302.02605 ↗First posted 6 Feb 2023cs.LG stat.MLcited by 12

Not on the map. Why?

Abstract (arXiv)

Recent studies indicate that kernel machines can often perform similarly or better than deep neural networks (DNNs) on small datasets. The interest in kernel machines has been additionally bolstered by the discovery of their equivalence to wide neural networks in certain regimes. However, a key feature of DNNs is their ability to scale the model size and training data size independently, whereas in traditional kernel machines model size is tied to data size. Because of this coupling, scaling kernel machines to large data has been computationally challenging. In this paper, we provide a way forward for constructing large-scale general kernel models, which are a generalization of kernel machines that decouples the model and data, allowing training on large datasets. Specifically, we introduce EigenPro 3.0, an algorithm based on projected dual preconditioned SGD and show scaling to model and data sizes which have not been possible with existing kernel methods.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 23 of its 68 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

23 of 68 references.

References · 23

  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  5. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  6. 2017WebVision Database: Visual Learning and Understanding from Web Datacited by 262
  7. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 175
  8. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  9. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  10. 2016GPflow: A Gaussian process library using TensorFlowcited by 139
  11. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  12. 2020Kernel methods through the roof: handling billions of points efficientlycited by 88
11 more
  1. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  2. 2017FALKON: An Optimal Large Scale Kernel Methodcited by 82
  3. 2019Enhanced Convolutional Neural Tangent Kernelscited by 76
  4. 2020On the Similarity between the Laplace and Neural Tangent Kernelscited by 76
  5. 2020Neural Kernels Without Tangentscited by 56
  6. 2018Product Kernel Interpolation for Scalable Gaussian Processescited by 31
  7. 2022Randomly pivoted Cholesky: Practical approximation of a kernel matrix with few entry evaluationscited by 31
  8. 2017Diving into the shallows: a computational perspective on large-scale shallow learningcited by 30
  9. 2021Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networkscited by 17
  10. 2015NYTRO: When Subsampling Meets Early Stoppingcited by 6
  11. 2022Transition to Linearity of General Neural Networks with Directed Acyclic Graph Architecturecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 5; 2026: 1.

12 papers.

  1. 2023Estimating Koopman operators with sketching to provably learn large scale dynamical systemscited by 15
  2. 2025xRFM: Accurate, scalable, and interpretable feature learning models for tabular datacited by 14
  3. 2023Improved Bayesian Regret Bounds for Thompson Sampling in Reinforcement Learningcited by 7
  4. 2024Optimal Kernel Quantile Learning with Random Featurescited by 4
  5. 2023On the Nystrom Approximation for Preconditioning in Kernel Machinescited by 2
  6. 2024Learning Analysis of Kernel Ridgeless Regression with Asymmetric Kernel Learningcited by 2
  7. 2024Fast training of large kernel models with delayed projectionscited by 2
  8. 2025Feature maps for the Laplacian kernel and its generalizationscited by 1
  9. 2025Faster Low-Rank Approximation and Kernel Ridge Regression via the Block-Nystr"om Methodcited by 1
  10. 2025Interpretable Self-Supervised Learning via Representer Landmarks and Nystr"om Approximationcited by 1
  11. 2025Joker: Joint Optimization Framework for Lightweight Kernel Machinescited by 0
  12. 2026Scalable Pairwise Kernel Learning with Stochastic Vec Trickcited by 0