# Dominant motion identification of multi-particle system using deep learning from video

arXiv: https://arxiv.org/abs/2104.12722 · HTML: https://mlsanity.com/paper/2104.12722/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 71 references that are in the map.
- First posted on arXiv 26 Apr 2021.

## Builds on (most cited first, 22 of 22)

- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10561/) (2017)
- [Gate-Variants of Gated Recurrent Unit (GRU) Neural Networks](https://mlsanity.com/paper/1701.05923/) (2017)
- [PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network](https://mlsanity.com/paper/1812.04426/) (2018)
- [Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1711.10566/) (2017)
- [Data-Driven Forecasting of High-Dimensional Chaotic Systems with Long Short-Term Memory Networks](https://mlsanity.com/paper/1802.07486/) (2018)
- [Machine Learning of Linear Differential Equations using Gaussian Processes](https://mlsanity.com/paper/1701.02440/) (2017)
- [Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey](https://mlsanity.com/paper/1911.12116/) (2019)
- [Multistep Neural Networks for Data-driven Discovery of Nonlinear Dynamical Systems](https://mlsanity.com/paper/1801.01236/) (2018)
- [Numerical Gaussian Processes for Time-dependent and Non-linear Partial Differential Equations](https://mlsanity.com/paper/1703.10230/) (2017)
- [Spatio-temporal video autoencoder with differentiable memory](https://mlsanity.com/paper/1511.06309/) (2015)
- [A Hierarchical Neural Autoencoder for Paragraphs and Documents](https://mlsanity.com/paper/1506.01057/) (2015)
- [Newtonian Image Understanding: Unfolding the Dynamics of Objects in Static Images](https://mlsanity.com/paper/1511.04048/) (2015)
- [Time Series Forecasting Using LSTM Networks: A Symbolic Approach](https://mlsanity.com/paper/2003.05672/) (2020)
- [Single Image 3D Interpreter Network](https://mlsanity.com/paper/1604.08685/) (2016)
- [Robust LSTM-Autoencoders for Face De-Occlusion in the Wild](https://mlsanity.com/paper/1612.08534/) (2016)

## Built on by (most cited first, 1 of 1)

- [Vector Representations of Vessel Trees](https://mlsanity.com/paper/2506.11163/) (2025)

## Abstract (arXiv)

Identifying underlying governing equations and physical relevant information from high-dimensional observable data has always been a challenge in physical sciences. With the recent advances in sensing technology and available datasets, various machine learning techniques have made it possible to distill underlying mathematical models from sufficiently clean and usable datasets. However, most of these techniques rely on prior knowledge of the system and noise-free data obtained by simulation of physical system or by direct measurements of the signals. Hence, the inference obtained by using these techniques is often unreliable to be used in the real world where observed data is noisy and requires feature engineering to extract relevant features. In this work, we provide a deep-learning framework that extracts relevant information from real-world videos of highly stochastic systems, with no prior knowledge and distills the underlying governing equation representing the system. We demonstrate this approach on videos of confined multi-agent/particle systems of ants, termites, fishes as well as a simulated confined multi-particle system with elastic collision interactions. Furthermore, we explore how these seemingly diverse systems have predictable underlying behavior. In this study, we have used computer vision and motion tracking to extract spatial trajectories of individual agents/particles in a system, and by using LSTM VAE we projected these features on a low-dimensional latent space from which the underlying differential equation representing the data was extracted using SINDy framework.