# Less is More: Rethinking Few-Shot Learning and Recurrent Neural Nets

arXiv: https://arxiv.org/abs/2209.14267 · HTML: https://mlsanity.com/paper/2209.14267/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 61 references that are in the map.
- First posted on arXiv 28 Sep 2022.

## Builds on (most cited first, 9 of 9)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Learning in High Dimension Always Amounts to Extrapolation](https://mlsanity.com/paper/2110.09485/) (2021)
- [Convolutional Neural Networks Analyzed via Convolutional Sparse Coding](https://mlsanity.com/paper/1607.08194/) (2016)
- [Recovery and Generalization in Over-Realized Dictionary Learning](https://mlsanity.com/paper/2006.06179/) (2020)
- [Convolutional Sparse Coding Fast Approximation with Application to Seismic Reflectivity Estimation](https://mlsanity.com/paper/2106.15296/) (2021)

## Built on by (most cited first, 3 of 3)

- [Domain-Aware Few-Shot Learning for Optical Coherence Tomography Noise Reduction](https://mlsanity.com/paper/2306.08102/) (2023)
- [One-Shot Image Restoration](https://mlsanity.com/paper/2404.17426/) (2024)
- [Back to Basics: Fast Denoising Iterative Algorithm](https://mlsanity.com/paper/2311.06634/) (2023)

## Abstract (arXiv)

The statistical supervised learning framework assumes an input-output set with a joint probability distribution that is reliably represented by the training dataset. The learner is then required to output a prediction rule learned from the training dataset's input-output pairs. In this work, we provide meaningful insights into the asymptotic equipartition property (AEP) citepShannon:1948 in the context of machine learning, and illuminate some of its potential ramifications for few-shot learning. We provide theoretical guarantees for reliable learning under the information-theoretic AEP, and for the generalization error with respect to the sample size. We then focus on a highly efficient recurrent neural net (RNN) framework and propose a reduced-entropy algorithm for few-shot learning. We also propose a mathematical intuition for the RNN as an approximation of a sparse coding solver. We verify the applicability, robustness, and computational efficiency of the proposed approach with image deblurring and optical coherence tomography (OCT) speckle suppression. Our experimental results demonstrate significant potential for improving learning models' sample efficiency, generalization, and time complexity, that can therefore be leveraged for practical real-time applications.