# Feature Embedding by Template Matching as a ResNet Block

arXiv: https://arxiv.org/abs/2210.00992 · HTML: https://mlsanity.com/paper/2210.00992/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 3 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Deeply-Supervised Nets](https://mlsanity.com/paper/1409.5185/) (2014)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations](https://mlsanity.com/paper/2106.01548/) (2021)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Interpreting Deep Visual Representations via Network Dissection](https://mlsanity.com/paper/1711.05611/) (2017)
- [Attributes2Classname: A discriminative model for attribute-based unsupervised zero-shot learning](https://mlsanity.com/paper/1705.01734/) (2017)

## Built on by (most cited first, 1 of 1)

- [Knowledge Distillation Layer that Lets the Student Decide](https://mlsanity.com/paper/2309.02843/) (2023)

## Abstract (arXiv)

Convolution blocks serve as local feature extractors and are the key to success of the neural networks. To make local semantic feature embedding rather explicit, we reformulate convolution blocks as feature selection according to the best matching kernel. In this manner, we show that typical ResNet blocks indeed perform local feature embedding via template matching once batch normalization (BN) followed by a rectified linear unit (ReLU) is interpreted as arg-max optimizer. Following this perspective, we tailor a residual block that explicitly forces semantically meaningful local feature embedding through using label information. Specifically, we assign a feature vector to each local region according to the classes that the corresponding region matches. We evaluate our method on three popular benchmark datasets with several architectures for image classification and consistently show that our approach substantially improves the performance of the baseline architectures.