# Learning recurrent representations for hierarchical behavior modeling

arXiv: https://arxiv.org/abs/1611.00094 · HTML: https://mlsanity.com/paper/1611.00094/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2016.

## Built on by (most cited first, 16 of 16)

- [The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions](https://mlsanity.com/paper/2104.02710/) (2021)
- [The Caltech Fish Counting Dataset: A Benchmark for Multiple-Object Tracking and Counting](https://mlsanity.com/paper/2207.09295/) (2022)
- [Self-Supervised Keypoint Discovery in Behavioral Videos](https://mlsanity.com/paper/2112.05121/) (2021)
- [BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos](https://mlsanity.com/paper/2212.07401/) (2022)
- [Unsupervised Learning of Neurosymbolic Encoders](https://mlsanity.com/paper/2107.13132/) (2021)
- [Learning interaction rules from multi-animal trajectories via augmented behavioral models](https://mlsanity.com/paper/2107.05326/) (2021)
- [Towards Fine-Grained Information: Identifying the Type and Location of Translation Errors](https://mlsanity.com/paper/2302.08975/) (2023)
- [A benchmark for computational analysis of animal behavior, using animal-borne tags](https://mlsanity.com/paper/2305.10740/) (2023)
- [MIMIC-MJX: Neuromechanical Emulation of Animal Behavior](https://mlsanity.com/paper/2511.20532/) (2025)
- [Imitation Learning of Factored Multi-agent Reactive Models](https://mlsanity.com/paper/1903.04714/) (2019)
- [Approximating the Manifold Structure of Attributed Incentive Salience from Large Scale Behavioural Data. A Representation Learning Approach Based on Artificial Neural Networks](https://mlsanity.com/paper/2108.01724/) (2021)
- [Overcoming the Domain Gap in Neural Action Representations](https://mlsanity.com/paper/2112.01176/) (2021)
- [Limitations of a proposed correction for slow drifts in decision criterion](https://mlsanity.com/paper/2205.10912/) (2022)
- [Review on Social Behavior Analysis of Laboratory Animals: From Methodologies to Applications](https://mlsanity.com/paper/2206.12651/) (2022)
- [Data-driven simulator of multi-animal behavior with unknown dynamics via offline and online reinforcement learning](https://mlsanity.com/paper/2510.10451/) (2025)
- [Modeling Fly Social Behavior with Agent-Centric Pose Forecasting](https://mlsanity.com/paper/2607.19548/) (2026)

## Abstract (arXiv)

We propose a framework for detecting action patterns from motion sequences and modeling the sensory-motor relationship of animals, using a generative recurrent neural network. The network has a discriminative part (classifying actions) and a generative part (predicting motion), whose recurrent cells are laterally connected, allowing higher levels of the network to represent high level phenomena. We test our framework on two types of data, fruit fly behavior and online handwriting. Our results show that 1) taking advantage of unlabeled sequences, by predicting future motion, significantly improves action detection performance when training labels are scarce, 2) the network learns to represent high level phenomena such as writer identity and fly gender, without supervision, and 3) simulated motion trajectories, generated by treating motion prediction as input to the network, look realistic and may be used to qualitatively evaluate whether the model has learnt generative control rules.