# Working memory inspired hierarchical video decomposition with transformative representations

arXiv: https://arxiv.org/abs/2204.10105 · HTML: https://mlsanity.com/paper/2204.10105/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 100 references that are in the map.
- First posted on arXiv 21 Apr 2022.

## Builds on (most cited first, 30 of 31)

- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI](https://mlsanity.com/paper/1907.07374/) (2019)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview](https://mlsanity.com/paper/1809.09573/) (2018)
- [Tensor Robust Principal Component Analysis with A New Tensor Nuclear Norm](https://mlsanity.com/paper/1804.03728/) (2018)
- [FISTA-Net: Learning A Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging](https://mlsanity.com/paper/2008.02683/) (2020)
- [Deep Unfolded Robust PCA with Application to Clutter Suppression in Ultrasound](https://mlsanity.com/paper/1811.08252/) (2018)
- [Reconstructive Sequence-Graph Network for Video Summarization](https://mlsanity.com/paper/2105.04066/) (2021)
- [Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection](https://mlsanity.com/paper/2110.05649/) (2021)
- [Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation](https://mlsanity.com/paper/1811.05255/) (2018)
- [Deep Back-Projection Networks for Single Image Super-resolution](https://mlsanity.com/paper/1904.05677/) (2019)
- [Sequential vessel segmentation via deep channel attention network](https://mlsanity.com/paper/2102.05229/) (2021)
- [Decomposition into Low-rank plus Additive Matrices for Background/Foreground Separation: A Review for a Comparative Evaluation with a Large-Scale Dataset](https://mlsanity.com/paper/1511.01245/) (2015)
- [Multimodal Deep Unfolding for Guided Image Super-Resolution](https://mlsanity.com/paper/2001.07575/) (2020)
- [Learning Mixtures of Low-Rank Models](https://mlsanity.com/paper/2009.11282/) (2020)
- [Robust Online Matrix Factorization for Dynamic Background Subtraction](https://mlsanity.com/paper/1705.10000/) (2017)
- [Learning-Based Algorithms for Vessel Tracking: A Review](https://mlsanity.com/paper/2012.08929/) (2020)
- [Working Memory Connections for LSTM](https://mlsanity.com/paper/2109.00020/) (2021)
- [Explaining Motion Relevance for Activity Recognition in Video Deep Learning Models](https://mlsanity.com/paper/2003.14285/) (2020)
- [Panoramic Robust PCA for Foreground-Background Separation on Noisy, Free-Motion Camera Video](https://mlsanity.com/paper/1712.06229/) (2017)
- [Robust PCA Unrolling Network for Super-resolution Vessel Extraction in X-ray Coronary Angiography](https://mlsanity.com/paper/2204.08466/) (2022)
- [Spatio-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data](https://mlsanity.com/paper/1902.08716/) (2019)
- [Hyper RPCA: Joint Maximum Correntropy Criterion and Laplacian Scale Mixture Modeling On-the-Fly for Moving Object Detection](https://mlsanity.com/paper/2006.07795/) (2020)
- [Efficient GPU implementation of randomized SVD and its applications](https://mlsanity.com/paper/2110.03423/) (2021)
- [Texture variation adaptive image denoising with nonlocal PCA](https://mlsanity.com/paper/1810.11282/) (2018)
- [A Deep Joint Sparse Non-negative Matrix Factorization Framework for Identifying the Common and Subject-specific Functional Units of Tongue Motion During Speech](https://mlsanity.com/paper/2007.04865/) (2020)
- [Deep learning with 4D spatio-temporal data representations for OCT-based force estimation](https://mlsanity.com/paper/2005.10033/) (2020)
- [Fast Robust Subspace Tracking via PCA in Sparse Data-Dependent Noise](https://mlsanity.com/paper/2006.08030/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)

## Abstract (arXiv)

Video decomposition is very important to extract moving foreground objects from complex backgrounds in computer vision, machine learning, and medical imaging, e.g., extracting moving contrast-filled vessels from the complex and noisy backgrounds of X-ray coronary angiography (XCA). However, the challenges caused by dynamic backgrounds, overlapping heterogeneous environments and complex noises still exist in video decomposition. To solve these problems, this study is the first to introduce a flexible visual working memory model in video decomposition tasks to provide interpretable and high-performance hierarchical deep architecture, integrating the transformative representations between sensory and control layers from the perspective of visual and cognitive neuroscience. Specifically, robust PCA unrolling networks acting as a structure-regularized sensor layer decompose XCA into sparse/low-rank structured representations to separate moving contrast-filled vessels from noisy and complex backgrounds. Then, patch recurrent convolutional LSTM networks with a backprojection module embody unstructured random representations of the control layer in working memory, recurrently projecting spatiotemporally decomposed nonlocal patches into orthogonal subspaces for heterogeneous vessel retrieval and interference suppression. This video decomposition deep architecture effectively restores the heterogeneous profiles of intensity and the geometries of moving objects against the complex background interferences. Experiments show that the proposed method significantly outperforms state-of-the-art methods in accurate moving contrast-filled vessel extraction with excellent flexibility and computational efficiency.