# Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli

arXiv: https://arxiv.org/abs/2411.01505 · HTML: https://mlsanity.com/paper/2411.01505/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 3 Nov 2024.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation](https://mlsanity.com/paper/1809.05571/) (2018)
- [Continuously Differentiable Exponential Linear Units](https://mlsanity.com/paper/1704.07483/) (2017)
- [It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos](https://mlsanity.com/paper/1604.00136/) (2016)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation](https://mlsanity.com/paper/2011.11630/) (2020)
- [Towards Segmenting Anything That Moves](https://mlsanity.com/paper/1902.03715/) (2019)
- [The neuroconnectionist research programme](https://mlsanity.com/paper/2209.03718/) (2022)
- [Learning to Segment Moving Objects](https://mlsanity.com/paper/1712.01127/) (2017)
- [Unsupervised Object Learning via Common Fate](https://mlsanity.com/paper/2110.06562/) (2021)
- [A Perturbation-Constrained Adversarial Attack for Evaluating the Robustness of Optical Flow](https://mlsanity.com/paper/2203.13214/) (2022)
- [Appearance-Based Refinement for Object-Centric Motion Segmentation](https://mlsanity.com/paper/2312.11463/) (2023)
- [Attacking Motion Estimation with Adversarial Snow](https://mlsanity.com/paper/2210.11242/) (2022)
- [Infinite dSprites for Disentangled Continual Learning: Separating Memory Edits from Generalization](https://mlsanity.com/paper/2312.16731/) (2023)

## Built on by (most cited first, 5 of 5)

- [Time Blindness: Why Video-Language Models Can't See What Humans Can?](https://mlsanity.com/paper/2505.24867/) (2025)
- [Do vision models perceive illusory motion in static images like humans?](https://mlsanity.com/paper/2604.09853/) (2026)
- [Appearance-free Action Recognition: Zero-shot Generalization in Humans and a Two-Pathway Model](https://mlsanity.com/paper/2604.16675/) (2026)
- [GenMatter: Perceiving Physical Objects with Generative Matter Models](https://mlsanity.com/paper/2604.22160/) (2026)
- [Vision Transformers Learn Gestalt-Like Figure-Ground Cues from Natural Images](https://mlsanity.com/paper/2607.08932/) (2026)

## Abstract (arXiv)

Humans excel at detecting and segmenting moving objects according to the Gestalt principle of "common fate". Remarkably, previous works have shown that human perception generalizes this principle in a zero-shot fashion to unseen textures or random dots. In this work, we seek to better understand the computational basis for this capability by evaluating a broad range of optical flow models and a neuroscience inspired motion energy model for zero-shot figure-ground segmentation of random dot stimuli. Specifically, we use the extensively validated motion energy model proposed by Simoncelli and Heeger in 1998 which is fitted to neural recordings in cortex area MT. We find that a cross section of 40 deep optical flow models trained on different datasets struggle to estimate motion patterns in random dot videos, resulting in poor figure-ground segmentation performance. Conversely, the neuroscience-inspired model significantly outperforms all optical flow models on this task. For a direct comparison to human perception, we conduct a psychophysical study using a shape identification task as a proxy to measure human segmentation performance. All state-of-the-art optical flow models fall short of human performance, but only the motion energy model matches human capability. This neuroscience-inspired model successfully addresses the lack of human-like zero-shot generalization to random dot stimuli in current computer vision models, and thus establishes a compelling link between the Gestalt psychology of human object perception and cortical motion processing in the brain. Code, models and datasets are available at https://github.com/mtangemann/motion_energy_segmentation