# Informative regularization for a multi-layer perceptron RR Lyrae classifier under data shift

arXiv: https://arxiv.org/abs/2303.06544 · HTML: https://mlsanity.com/paper/2303.06544/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 83 references that are in the map.
- First posted on arXiv 12 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems](https://mlsanity.com/paper/1903.12394/) (2019)
- [Priors in Bayesian Deep Learning: A Review](https://mlsanity.com/paper/2105.06868/) (2021)
- [Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey](https://mlsanity.com/paper/2005.10691/) (2020)
- [Scalable End-to-end Recurrent Neural Network for Variable star classification](https://mlsanity.com/paper/2002.00994/) (2020)
- [Deep multi-survey classification of variable stars](https://mlsanity.com/paper/1810.09440/) (2018)
- [Classification of Periodic Variable Stars with Novel Cyclic-Permutation Invariant Neural Networks](https://mlsanity.com/paper/2011.01243/) (2020)
- [Supervised detection of anomalous light-curves in massive astronomical catalogs](https://mlsanity.com/paper/1404.4888/) (2014)
- [Clustering Based Feature Learning on Variable Stars](https://mlsanity.com/paper/1602.08977/) (2016)
- [An improved quasar detection method in EROS-2 and MACHO LMC datasets](https://mlsanity.com/paper/1304.0401/) (2013)
- [Disease Prediction based on Functional Connectomes using a Scalable and Spatially-Informed Support Vector Machine](https://mlsanity.com/paper/1310.5415/) (2013)
- [Constraint-Based Regularization of Neural Networks](https://mlsanity.com/paper/2006.10114/) (2020)
- [From FATS to feets: Further improvements to an astronomical feature extraction tool based on machine learning](https://mlsanity.com/paper/1809.02154/) (2018)
- [Automatic classification of eclipsing binary stars using deep learning methods](https://mlsanity.com/paper/2108.01640/) (2021)
- [Informative Bayesian model selection for RR Lyrae star classifiers](https://mlsanity.com/paper/2105.11531/) (2021)

## Abstract (arXiv)

In recent decades, machine learning has provided valuable models and algorithms for processing and extracting knowledge from time-series surveys. Different classifiers have been proposed and performed to an excellent standard. Nevertheless, few papers have tackled the data shift problem in labeled training sets, which occurs when there is a mismatch between the data distribution in the training set and the testing set. This drawback can damage the prediction performance in unseen data. Consequently, we propose a scalable and easily adaptable approach based on an informative regularization and an ad-hoc training procedure to mitigate the shift problem during the training of a multi-layer perceptron for RR Lyrae classification. We collect ranges for characteristic features to construct a symbolic representation of prior knowledge, which was used to model the informative regularizer component. Simultaneously, we design a two-step back-propagation algorithm to integrate this knowledge into the neural network, whereby one step is applied in each epoch to minimize classification error, while another is applied to ensure regularization. Our algorithm defines a subset of parameters (a mask) for each loss function. This approach handles the forgetting effect, which stems from a trade-off between these loss functions (learning from data versus learning expert knowledge) during training. Experiments were conducted using recently proposed shifted benchmark sets for RR Lyrae stars, outperforming baseline models by up to 3% through a more reliable classifier. Our method provides a new path to incorporate knowledge from characteristic features into artificial neural networks to manage the underlying data shift problem.