# Kernel function impact on convolutional neural networks

arXiv: https://arxiv.org/abs/2302.10266 · HTML: https://mlsanity.com/paper/2302.10266/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 55 references that are in the map.
- First posted on arXiv 20 Feb 2023.

## Builds on (most cited first, 27 of 27)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Challenges in Representation Learning: A report on three machine learning contests](https://mlsanity.com/paper/1307.0414/) (2013)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://mlsanity.com/paper/2003.00392/) (2020)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [Compact Bilinear Pooling](https://mlsanity.com/paper/1511.06062/) (2015)
- [Learning Attentive Pairwise Interaction for Fine-Grained Classification](https://mlsanity.com/paper/2002.10191/) (2020)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [Global Second-order Pooling Convolutional Networks](https://mlsanity.com/paper/1811.12006/) (2018)
- [From Facial Expression Recognition to Interpersonal Relation Prediction](https://mlsanity.com/paper/1609.06426/) (2016)
- [Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1909.11378/) (2019)
- [Channel Interaction Networks for Fine-Grained Image Categorization](https://mlsanity.com/paper/2003.05235/) (2020)
- [Deep Learning using Linear Support Vector Machines](https://mlsanity.com/paper/1306.0239/) (2013)
- [Non-linear Convolution Filters for CNN-based Learning](https://mlsanity.com/paper/1708.07038/) (2017)
- [Covariance Pooling For Facial Expression Recognition](https://mlsanity.com/paper/1805.04855/) (2018)
- [Revisiting Pose-Normalization for Fine-Grained Few-Shot Recognition](https://mlsanity.com/paper/2004.00705/) (2020)
- [Improved Bilinear Pooling with CNNs](https://mlsanity.com/paper/1707.06772/) (2017)
- [Kervolutional Neural Networks](https://mlsanity.com/paper/1904.03955/) (2019)
- [SchiNet: Automatic Estimation of Symptoms of Schizophrenia from Facial Behaviour Analysis](https://mlsanity.com/paper/1808.02531/) (2018)
- [TextCaps : Handwritten Character Recognition with Very Small Datasets](https://mlsanity.com/paper/1904.08095/) (2019)
- [Expression Analysis Based on Face Regions in Read-world Conditions](https://mlsanity.com/paper/1911.05188/) (2019)
- [Stochastic Optimization of Plain Convolutional Neural Networks with Simple methods](https://mlsanity.com/paper/2001.08856/) (2020)
- [Kernelized dense layers for facial expression recognition](https://mlsanity.com/paper/2009.10814/) (2020)
- [Universal Pooling -- A New Pooling Method for Convolutional Neural Networks](https://mlsanity.com/paper/1907.11440/) (2019)

## Built on by (most cited first, 2 of 2)

- [Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networks](https://mlsanity.com/paper/2312.15229/) (2023)
- [IIKL: Isometric Immersion Kernel Learning with Riemannian Manifold for Geometric Preservation](https://mlsanity.com/paper/2505.06288/) (2025)

## Abstract (arXiv)

This paper investigates the usage of kernel functions at the different layers in a convolutional neural network. We carry out extensive studies of their impact on convolutional, pooling and fully-connected layers. We notice that the linear kernel may not be sufficiently effective to fit the input data distributions, whereas high order kernels prone to over-fitting. This leads to conclude that a trade-off between complexity and performance should be reached. We show how one can effectively leverage kernel functions, by introducing a more distortion aware pooling layers which reduces over-fitting while keeping track of the majority of the information fed into subsequent layers. We further propose Kernelized Dense Layers (KDL), which replace fully-connected layers, and capture higher order feature interactions. The experiments on conventional classification datasets i.e. MNIST, FASHION-MNIST and CIFAR-10, show that the proposed techniques improve the performance of the network compared to classical convolution, pooling and fully connected layers. Moreover, experiments on fine-grained classification i.e. facial expression databases, namely RAF-DB, FER2013 and ExpW demonstrate that the discriminative power of the network is boosted, since the proposed techniques improve the awareness to slight visual details and allows the network reaching state-of-the-art results.