# Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layers

arXiv: https://arxiv.org/abs/2010.08012 · HTML: https://mlsanity.com/paper/2010.08012/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 3 of 3)

- [Neural Production Systems: Learning Rule-Governed Visual Dynamics](https://mlsanity.com/paper/2103.01937/) (2021)
- [Transformers with Competitive Ensembles of Independent Mechanisms](https://mlsanity.com/paper/2103.00336/) (2021)
- [CNT (Conditioning on Noisy Targets): A new Algorithm for Leveraging Top-Down Feedback](https://mlsanity.com/paper/2210.09505/) (2022)

## Abstract (arXiv)

Feed-forward neural networks consist of a sequence of layers, in which each layer performs some processing on the information from the previous layer. A downside to this approach is that each layer (or module, as multiple modules can operate in parallel) is tasked with processing the entire hidden state, rather than a particular part of the state which is most relevant for that module. Methods which only operate on a small number of input variables are an essential part of most programming languages, and they allow for improved modularity and code re-usability. Our proposed method, Neural Function Modules (NFM), aims to introduce the same structural capability into deep learning. Most of the work in the context of feed-forward networks combining top-down and bottom-up feedback is limited to classification problems. The key contribution of our work is to combine attention, sparsity, top-down and bottom-up feedback, in a flexible algorithm which, as we show, improves the results in standard classification, out-of-domain generalization, generative modeling, and learning representations in the context of reinforcement learning.