# Task-Driven Convolutional Recurrent Models of the Visual System

arXiv: https://arxiv.org/abs/1807.00053 · HTML: https://mlsanity.com/paper/1807.00053/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Jun 2018.

## Built on by (most cited first, 27 of 27)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [From Language to Cognition: How LLMs Outgrow the Human Language Network](https://mlsanity.com/paper/2503.01830/) (2025)
- [The least-control principle for local learning at equilibrium](https://mlsanity.com/paper/2207.01332/) (2022)
- [On-Off Center-Surround Receptive Fields for Accurate and Robust Image Classification](https://mlsanity.com/paper/2106.07091/) (2021)
- [Scaling Laws for Task-Optimized Models of the Primate Visual Ventral Stream](https://mlsanity.com/paper/2411.05712/) (2024)
- [Brain-Model Evaluations Need the NeuroAI Turing Test](https://mlsanity.com/paper/2502.16238/) (2025)
- [Differentiable Optimization of Similarity Scores Between Models and Brains](https://mlsanity.com/paper/2407.07059/) (2024)
- [Better artificial intelligence does not mean better models of biology](https://mlsanity.com/paper/2504.16940/) (2025)
- [Synaptic Weight Distributions Depend on the Geometry of Plasticity](https://mlsanity.com/paper/2305.19394/) (2023)
- [Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning](https://mlsanity.com/paper/2110.09374/) (2021)
- [Visual Attention Emerges from Recurrent Sparse Reconstruction](https://mlsanity.com/paper/2204.10962/) (2022)
- [On the role of feedback in visual processing: a predictive coding perspective](https://mlsanity.com/paper/2106.04225/) (2021)
- [Category-orthogonal object features guide information processing in recurrent neural networks trained for object categorization](https://mlsanity.com/paper/2111.07898/) (2021)
- [IB-U-Nets: Improving medical image segmentation tasks with 3D Inductive Biased kernels](https://mlsanity.com/paper/2210.15949/) (2022)
- [What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty](https://mlsanity.com/paper/2603.02491/) (2026)
- [Multi-Agent Feedback Enabled Neural Networks for Intelligent Communications](https://mlsanity.com/paper/2205.10750/) (2022)
- [Beyond the limitations of any imaginable mechanism: large language models and psycholinguistics](https://mlsanity.com/paper/2303.00077/) (2023)
- [A Framework for Standardizing Similarity Measures in a Rapidly Evolving Field](https://mlsanity.com/paper/2409.18333/) (2024)
- [Improving Unsupervised Task-driven Models of Ventral Visual Stream via Relative Position Predictivity](https://mlsanity.com/paper/2505.08316/) (2025)
- [Task-Optimized Convolutional Recurrent Networks Align with Tactile Processing in the Rodent Brain](https://mlsanity.com/paper/2505.18361/) (2025)
- [NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity](https://mlsanity.com/paper/2607.02317/) (2026)
- [Fixing the problems of deep neural networks will require better training data and learning algorithms](https://mlsanity.com/paper/2311.12819/) (2023)
- [Self-Attention-Based Contextual Modulation Improves Neural System Identification](https://mlsanity.com/paper/2406.07843/) (2024)
- [Recurrent Joint Embedding Predictive Architecture with Recurrent Forward Propagation Learning](https://mlsanity.com/paper/2411.16695/) (2024)
- [Sensory robustness through top-down feedback and neural stochasticity in recurrent vision models](https://mlsanity.com/paper/2508.07115/) (2025)
- [Self-organized MT Direction Maps Emerge from Spatiotemporal Contrastive Optimization](https://mlsanity.com/paper/2605.11718/) (2026)

## Abstract (arXiv)

Feed-forward convolutional neural networks (CNNs) are currently state-of-the-art for object classification tasks such as ImageNet. Further, they are quantitatively accurate models of temporally-averaged responses of neurons in the primate brain's visual system. However, biological visual systems have two ubiquitous architectural features not shared with typical CNNs: local recurrence within cortical areas, and long-range feedback from downstream areas to upstream areas. Here we explored the role of recurrence in improving classification performance. We found that standard forms of recurrence (vanilla RNNs and LSTMs) do not perform well within deep CNNs on the ImageNet task. In contrast, novel cells that incorporated two structural features, bypassing and gating, were able to boost task accuracy substantially. We extended these design principles in an automated search over thousands of model architectures, which identified novel local recurrent cells and long-range feedback connections useful for object recognition. Moreover, these task-optimized ConvRNNs matched the dynamics of neural activity in the primate visual system better than feedforward networks, suggesting a role for the brain's recurrent connections in performing difficult visual behaviors.