# A Memory Network Approach for Story-based Temporal Summarization of 360deg Videos

arXiv: https://arxiv.org/abs/1805.02838 · HTML: https://mlsanity.com/paper/1805.02838/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2018.

## Built on by (most cited first, 29 of 29)

- [MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding](https://mlsanity.com/paper/2404.05726/) (2024)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory](https://mlsanity.com/paper/2211.00924/) (2022)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Retrieval-Enhanced Machine Learning](https://mlsanity.com/paper/2205.01230/) (2022)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [SQUID: Deep Feature In-Painting for Unsupervised Anomaly Detection](https://mlsanity.com/paper/2111.13495/) (2021)
- [Memories are One-to-Many Mapping Alleviators in Talking Face Generation](https://mlsanity.com/paper/2212.05005/) (2022)
- [Panoramic Vision Transformer for Saliency Detection in 360deg Videos](https://mlsanity.com/paper/2209.08956/) (2022)
- [Exploiting Structural Consistency of Chest Anatomy for Unsupervised Anomaly Detection in Radiography Images](https://mlsanity.com/paper/2403.08689/) (2024)
- [Unsupervised Outlier Detection using Memory and Contrastive Learning](https://mlsanity.com/paper/2107.12642/) (2021)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Memory-based Semantic Segmentation for Off-road Unstructured Natural Environments](https://mlsanity.com/paper/2108.05635/) (2021)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [Spherical World-Locking for Audio-Visual Localization in Egocentric Videos](https://mlsanity.com/paper/2408.05364/) (2024)
- [OmniPD: One-Step Person Detection in Top-View Omnidirectional Indoor Scenes](https://mlsanity.com/paper/2204.06846/) (2022)
- [An Integrated System for Spatio-Temporal Summarization of 360-degrees Videos](https://mlsanity.com/paper/2312.02576/) (2023)
- [REEF: Relevance-Aware and Efficient LLM Adapter for Video Understanding](https://mlsanity.com/paper/2504.05491/) (2025)
- [Panoramic Image-to-Image Translation](https://mlsanity.com/paper/2304.04960/) (2023)
- [WinDB: HMD-free and Distortion-free Panoptic Video Fixation Learning](https://mlsanity.com/paper/2305.13901/) (2023)
- [CNN with large memory layers](https://mlsanity.com/paper/2101.11685/) (2021)
- [Adaptive Memory Management for Video Object Segmentation](https://mlsanity.com/paper/2204.06626/) (2022)
- [Saliency-based Multiple Region of Interest Detection from a Single 360deg image](https://mlsanity.com/paper/2209.03656/) (2022)
- [FindView: Precise Target View Localization Task for Look Around Agents](https://mlsanity.com/paper/2303.09054/) (2023)
- [A Human-Annotated Video Dataset for Training and Evaluation of 360-Degree Video Summarization Methods](https://mlsanity.com/paper/2406.02991/) (2024)
- [TSalV360: A Method and Dataset for Text-driven Saliency Detection in 360-Degrees Videos](https://mlsanity.com/paper/2509.26208/) (2025)

## Abstract (arXiv)

We address the problem of story-based temporal summarization of long 360deg videos. We propose a novel memory network model named Past-Future Memory Network (PFMN), in which we first compute the scores of 81 normal field of view (NFOV) region proposals cropped from the input 360deg video, and then recover a latent, collective summary using the network with two external memories that store the embeddings of previously selected subshots and future candidate subshots. Our major contributions are two-fold. First, our work is the first to address story-based temporal summarization of 360deg videos. Second, our model is the first attempt to leverage memory networks for video summarization tasks. For evaluation, we perform three sets of experiments. First, we investigate the view selection capability of our model on the Pano2Vid dataset. Second, we evaluate the temporal summarization with a newly collected 360deg video dataset. Finally, we experiment our model's performance in another domain, with image-based storytelling VIST dataset. We verify that our model achieves state-of-the-art performance on all the tasks.