Adjusting Model Size in Continual Gaussian Processes: How Big is Big Enough?
Not on the map. Why?
Abstract (arXiv)
Many machine learning models require setting a parameter that controls their size before training, e.g. number of neurons in DNNs, or inducing points in GPs. Increasing capacity typically improves performance until all the information from the dataset is captured. After this point, computational cost keeps increasing, without improved performance. This leads to the question "How big is big enough?" We investigate this problem for Gaussian processes (single-layer neural networks) in continual learning. Here, data becomes available incrementally, and the final dataset size will therefore not be known before training, preventing the use of heuristics for setting a fixed model size. We develop a method to automatically adjust model size while maintaining near-optimal performance. Our experimental procedure follows the constraint that any hyperparameters must be set without seeing dataset properties, and we show that our method performs well across diverse datasets without the need to adjust its hyperparameter, showing it requires less tuning than others.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 14 Aug 2024.
Builds on
30 of 50 references.
References · 30
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
- 2016Learning without Forgettingcited by 2,479
- 2017Gradient Episodic Memory for Continual Learningcited by 1,882
- 2016Progressive Neural Networkscited by 1,347
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 718
- 2017Lifelong Learning with Dynamically Expandable Networkscited by 692
- 2018Progress & Compress: A scalable framework for continual learningcited by 516
- 2013Gaussian Processes for Big Datacited by 380
- 2017Variational Continual Learningcited by 335
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 316
- 2018Towards Robust Evaluations of Continual Learningcited by 148
- 2016GPflow: A Gaussian process library using TensorFlowcited by 139
18 more
- 2016Understanding Probabilistic Sparse Gaussian Process Approximationscited by 68
- 2019Rates of Convergence for Sparse Variational Gaussian Process Regressioncited by 62
- 2015On Sparse variational methods and the Kullback-Leibler divergence between stochastic processescited by 56
- 2020Convergence of Sparse Variational Inference in Gaussian Processes Regressioncited by 53
- 2020Sparse Gaussian Processes with Spherical Harmonic Featurescited by 43
- 2017Streaming Sparse Gaussian Process Approximationscited by 41
- 2023Continual Learning via Sequential Function-Space Variational Inferencecited by 29
- 2015Modeling and interpolation of the ambient magnetic field by Gaussian processescited by 28
- 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
- 2020Variational Auto-Regressive Gaussian Processes for Continual Learningcited by 23
- 2021Conditioning Sparse Variational Gaussian Processes for Online Decision-makingcited by 23
- 2023Low-rank extended Kalman filtering for online learning of neural networks from streaming datacited by 23
- 2022When, where, and how to add new neurons to ANNscited by 18
- 2023Self-Expanding Neural Networkscited by 14
- 2019Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learningcited by 11
- 2023Memory-Based Dual Gaussian Processes for Sequential Learningcited by 11
- 2021Adaptive Inducing Points Selection For Gaussian Processescited by 8
- 2018Fully Scalable Gaussian Processes using Subspace Inducing Inputscited by 4
Built on by
Citing papers per year: 2024: 1.
1 paper.