# Unsupervised Learning of Equivariant Structure from Sequences

arXiv: https://arxiv.org/abs/2210.05972 · HTML: https://mlsanity.com/paper/2210.05972/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 85 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Disentangling by Factorising](https://mlsanity.com/paper/1802.05983/) (2018)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style](https://mlsanity.com/paper/2106.04619/) (2021)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Conditional Object-Centric Learning from Video](https://mlsanity.com/paper/2111.12594/) (2021)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction](https://mlsanity.com/paper/2003.01460/) (2020)
- [MDP Homomorphic Networks: Group Symmetries in Reinforcement Learning](https://mlsanity.com/paper/2006.16908/) (2020)
- [Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding](https://mlsanity.com/paper/2007.10930/) (2020)

## Built on by (most cited first, 15 of 15)

- [GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers](https://mlsanity.com/paper/2310.10375/) (2023)
- [Neural Fourier Transform: A General Approach to Equivariant Representation Learning](https://mlsanity.com/paper/2305.18484/) (2023)
- [Set-based Neural Network Encoding Without Weight Tying](https://mlsanity.com/paper/2305.16625/) (2023)
- [Neural Isometries: Taming Transformations for Equivariant ML](https://mlsanity.com/paper/2405.19296/) (2024)
- [Retrieval-based Disentangled Representation Learning with Natural Language Supervision](https://mlsanity.com/paper/2212.07699/) (2022)
- [Enriching Disentanglement: From Logical Definitions to Quantitative Metrics](https://mlsanity.com/paper/2305.11512/) (2023)
- [Algebras of actions in an agent's representations of the world](https://mlsanity.com/paper/2310.01536/) (2023)
- [CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational AutoEncoder](https://mlsanity.com/paper/2401.08897/) (2024)
- [Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions](https://mlsanity.com/paper/2510.21706/) (2025)
- [Block-Diagonal Orthogonal Relation and Matrix Entity for Knowledge Graph Embedding](https://mlsanity.com/paper/2401.05967/) (2024)
- [Inter-environmental world modeling for continuous and compositional dynamics](https://mlsanity.com/paper/2503.09911/) (2025)
- [Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations](https://mlsanity.com/paper/2510.17313/) (2025)
- [KEEC: Koopman Embedded Equivariant Control](https://mlsanity.com/paper/2312.01544/) (2023)
- [Equivariant Latent Alignment via Flow Matching under Group Symmetries](https://mlsanity.com/paper/2605.30705/) (2026)
- [Blind Recovery of Latent Domains via Unsupervised Symmetry Discovery](https://mlsanity.com/paper/2606.17782/) (2026)

## Abstract (arXiv)

In this study, we present meta-sequential prediction (MSP), an unsupervised framework to learn the symmetry from the time sequence of length at least three. Our method leverages the stationary property (e.g. constant velocity, constant acceleration) of the time sequence to learn the underlying equivariant structure of the dataset by simply training the encoder-decoder model to be able to predict the future observations. We will demonstrate that, with our framework, the hidden disentangled structure of the dataset naturally emerges as a by-product by applying simultaneous block-diagonalization to the transition operators in the latent space, the procedure which is commonly used in representation theory to decompose the feature-space based on the type of response to group actions. We will showcase our method from both empirical and theoretical perspectives. Our result suggests that finding a simple structured relation and learning a model with extrapolation capability are two sides of the same coin. The code is available at https://github.com/takerum/meta_sequential_prediction.