Kernel-Based Ensemble Learning in Python

We propose a new supervised learning algorithm, for classification and regression problems where two or more preliminary predictors are available. We introduce textttKernelCobra, a non-linear learning strategy for combining an arbitrary number of initial predictors. textttKernelCobra builds on the COBRA algorithm introduced by citetbiau2016cobra, which combined estimators based on a notion of proximity of predictions on the training data. While the COBRA algorithm used a binary threshold to declare which training data were close and to be used, we generalize this idea by using a kernel to better encapsulate the proximity information. Such a smoothing kernel provides more representative weights to each of the training points which are used to build the aggregate and final predictor, and textttKernelCobra systematically outperforms the COBRA algorithm. While COBRA is intended for regression, textttKernelCobra deals with classification and regression. textttKernelCobra is included as part of the open source Python package textttPycobra (0.2.4 and onward), introduced by citetguedj2018pycobra. Numerical experiments assess the performance (in terms of pure prediction and computational complexity) of textttKernelCobra on real-life and synthetic datasets.