# Predictive Event Segmentation and Representation with Neural Networks: A Self-Supervised Model Assessed by Psychological Experiments

arXiv: https://arxiv.org/abs/2210.05710 · HTML: https://mlsanity.com/paper/2210.05710/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 138 references that are in the map.
- First posted on arXiv 4 Oct 2022.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Self-supervised Learning: Generative or Contrastive](https://mlsanity.com/paper/2006.08218/) (2020)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [Learning to Generate Reviews and Discovering Sentiment](https://mlsanity.com/paper/1704.01444/) (2017)
- [Self-supervised learning through the eyes of a child](https://mlsanity.com/paper/2007.16189/) (2020)
- [Deep Episodic Memory: Encoding, Recalling, and Predicting Episodic Experiences for Robot Action Execution](https://mlsanity.com/paper/1801.04134/) (2018)
- [Action Localization through Continual Predictive Learning](https://mlsanity.com/paper/2003.12185/) (2020)
- [Autonomous Identification and Goal-Directed Invocation of Event-Predictive Behavioral Primitives](https://mlsanity.com/paper/1902.09948/) (2019)
- [Time Perception: A Review on Psychological, Computational and Robotic Models](https://mlsanity.com/paper/2007.11845/) (2020)
- [Online recognition of unsegmented actions with hierarchical SOM architecture](https://mlsanity.com/paper/2104.11637/) (2021)
- [Similarities and differences between stimulus tuning in the inferotemporal visual cortex and convolutional networks](https://mlsanity.com/paper/1612.06975/) (2016)

## Built on by (most cited first, 1 of 1)

- [Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2601.15655/) (2026)

## Abstract (arXiv)

People segment complex, ever-changing and continuous experience into basic, stable and discrete spatio-temporal experience units, called events. Event segmentation literature investigates the mechanisms that allow people to extract events. Event segmentation theory points out that people predict ongoing activities and observe prediction error signals to find event boundaries that keep events apart. In this study, we investigated the mechanism giving rise to this ability by a computational model and accompanying psychological experiments. Inspired from event segmentation theory and predictive processing, we introduced a self-supervised model of event segmentation. This model consists of neural networks that predict the sensory signal in the next time-step to represent different events, and a cognitive model that regulates these networks on the basis of their prediction errors. In order to verify the ability of our model in segmenting events, learning them during passive observation, and representing them in its internal representational space, we prepared a video that depicts human behaviors represented by point-light displays. We compared event segmentation behaviors of participants and our model with this video in two hierarchical event segmentation levels. By using point-biserial correlation technique, we demonstrated that event segmentation decisions of our model correlated with the responses of participants. Moreover, by approximating representation space of participants by a similarity-based technique, we showed that our model formed a similar representation space with those of participants. The result suggests that our model that tracks the prediction error signals can produce human-like event boundaries and event representations. Finally, we discussed our contribution to the literature of event cognition and our understanding of how event segmentation is implemented in the brain.