# Learning long-term dependencies for action recognition with a biologically-inspired deep network

arXiv: https://arxiv.org/abs/1611.05216 · HTML: https://mlsanity.com/paper/1611.05216/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2016.

## Built on by (most cited first, 6 of 6)

- [Part Aware Contrastive Learning for Self-Supervised Action Recognition](https://mlsanity.com/paper/2305.00666/) (2023)
- [DSANet: Dynamic Segment Aggregation Network for Video-Level Representation Learning](https://mlsanity.com/paper/2105.12085/) (2021)
- [Deep Learning Computer Vision Algorithms for Real-time UAVs On-board Camera Image Processing](https://mlsanity.com/paper/2211.01037/) (2022)
- [Punching Bag vs. Punching Person: Motion Transferability in Videos](https://mlsanity.com/paper/2508.00085/) (2025)
- [A Survey on Human Action Recognition](https://mlsanity.com/paper/2301.06082/) (2022)
- [Texture-Based Input Feature Selection for Action Recognition](https://mlsanity.com/paper/2303.00138/) (2023)

## Abstract (arXiv)

Despite a lot of research efforts devoted in recent years, how to efficiently learn long-term dependencies from sequences still remains a pretty challenging task. As one of the key models for sequence learning, recurrent neural network (RNN) and its variants such as long short term memory (LSTM) and gated recurrent unit (GRU) are still not powerful enough in practice. One possible reason is that they have only feedforward connections, which is different from the biological neural system that is typically composed of both feedforward and feedback connections. To address this problem, this paper proposes a biologically-inspired deep network, called shuttleNetfootnoteOur code is available at urlhttps://github.com/shiyemin/shuttlenet. Technologically, the shuttleNet consists of several processors, each of which is a GRU while associated with multiple groups of cells and states. Unlike traditional RNNs, all processors inside shuttleNet are loop connected to mimic the brain's feedforward and feedback connections, in which they are shared across multiple pathways in the loop connection. Attention mechanism is then employed to select the best information flow pathway. Extensive experiments conducted on two benchmark datasets (i.e UCF101 and HMDB51) show that we can beat state-of-the-art methods by simply embedding shuttleNet into a CNN-RNN framework.