mlsanityData to 7 Oct 2026

Learning the Predictability of the Future

arXiv 2101.01600 ↗First posted 1 Jan 2021cs.CV cs.LG eess.IVcited by 48

Not on the map. Why?

Abstract (arXiv)

We introduce a framework for learning from unlabeled video what is predictable in the future. Instead of committing up front to features to predict, our approach learns from data which features are predictable. Based on the observation that hyperbolic geometry naturally and compactly encodes hierarchical structure, we propose a predictive model in hyperbolic space. When the model is most confident, it will predict at a concrete level of the hierarchy, but when the model is not confident, it learns to automatically select a higher level of abstraction. Experiments on two established datasets show the key role of hierarchical representations for action prediction. Although our representation is trained with unlabeled video, visualizations show that action hierarchies emerge in the representation.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 49 of its 81 references that are in the map. First posted on arXiv 1 Jan 2021.

Builds on

49 of 81 references.

References · 49

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  11. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  12. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
37 more
  1. 2019Hyperbolic Graph Convolutional Neural Networkscited by 400
  2. 2018Hyperbolic Neural Networkscited by 393
  3. 2016Generating Videos with Scene Dynamicscited by 376
  4. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  5. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  6. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  7. 2019DRIT++: Diverse Image-to-Image Translation via Disentangled Representationscited by 244
  8. 2019Hyperbolic Graph Neural Networkscited by 244
  9. 2020FineGym: A Hierarchical Video Dataset for Fine-grained Action Understandingcited by 232
  10. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  11. 2019Hyperbolic Image Embeddingscited by 219
  12. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  13. 2018Riemannian Adaptive Optimization Methodscited by 187
  14. 2019Video Representation Learning by Dense Predictive Codingcited by 175
  15. 2018Tracking Emerges by Colorizing Videoscited by 166
  16. 2018Hyperbolic Entailment Cones for Learning Hierarchical Embeddingscited by 160
  17. 2018Poincar'e GloVe: Hyperbolic Word Embeddingscited by 155
  18. 2019Temporal Cycle-Consistency Learningcited by 151
  19. 2020Memory-augmented Dense Predictive Coding for Video Representation Learningcited by 137
  20. 2018Hyperbolic Attention Networkscited by 122
  21. 2018Convolutional Sequence to Sequence Model for Human Dynamicscited by 115
  22. 2020Geoopt: Riemannian Optimization in PyTorchcited by 109
  23. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  24. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  25. 2014Video (language) modeling: a baseline for generative models of natural videoscited by 92
  26. 2019Self-supervised Spatio-temporal Representation Learning for Videos by Predicting Motion and Appearance Statisticscited by 92
  27. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  28. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86
  29. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  30. 2015Learning Visual Predictive Models of Physics for Playing Billiardscited by 58
  31. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50
  32. 2020Hierarchical Image Classification using Entailment Cone Embeddingscited by 46
  33. 2015Learning visual groups from co-occurrences in space and timecited by 42
  34. 2019Predicting the Future: A Jointly Learnt Model for Action Anticipationcited by 41
  35. 2019Learning Video Representations using Contrastive Bidirectional Transformercited by 39
  36. 2018Time-Agnostic Prediction: Predicting Predictable Video Framescited by 37
  37. 2017A hierarchical loss and its problems when classifying non-hierarchicallycited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 12; 2024: 10; 2025: 7; 2026: 2.

48 papers.

Sort:

By area:

  1. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  2. 2022Hyperbolic Image Segmentationcited by 101
  3. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 84
  4. 2023Hyperbolic Deep Learning in Computer Vision: A Surveycited by 72
  5. 2022Hyperbolic Contrastive Learning for Visual Representations beyond Objectscited by 58
  6. 2023HYperbolic Self-Paced Learning for Self-Supervised Skeleton-based Action Representationscited by 42
  7. 2021Hyperbolic Busemann Learning with Ideal Prototypescited by 40
  8. 2022Intention-Conditioned Long-Term Human Egocentric Action Forecastingcited by 35
  9. 2022Visual Abductive Reasoningcited by 30
  10. 2023Poincar'e ResNetcited by 24
  11. 2022The Euclidean Space is Evil: Hyperbolic Attribute Editing for Few-shot Image Generationcited by 22
  12. 2022Hierarchical Sketch Induction for Paraphrase Generationcited by 21
36 more
  1. 2024Differentiable Robot Renderingcited by 20
  2. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  3. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  4. 2023Hyperbolic Active Learning for Semantic Segmentation under Domain Shiftcited by 18
  5. 2023Holistic Representation Learning for Multitask Trajectory Anomaly Detectioncited by 15
  6. 2025Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometriescited by 15
  7. 2021HyperExpan: Taxonomy Expansion with Hyperbolic Representation Learningcited by 14
  8. 2022Are we certain it's anomalous?cited by 13
  9. 2025Heterogeneous Skeleton-Based Action Representation Learningcited by 12
  10. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  11. 2024Hyperbolic Learning with Multimodal Large Language Modelscited by 10
  12. 2023Hyperbolic Random Forestscited by 9
  13. 2024Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincar'e Ballcited by 9
  14. 2024Human Action Anticipation: A Surveycited by 8
  15. 2023Attributable and Scalable Opinion Summarizationcited by 6
  16. 2021Review of Video Predictive Understanding: Early Action Recognition and Future Action Predictioncited by 5
  17. 2022Non-linear Embeddings in Hilbert Simplex Geometrycited by 5
  18. 2022Representing Spatial Trajectories as Distributionscited by 5
  19. 2022FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observationscited by 5
  20. 2023Hierarchical Explanations for Video Action Recognitioncited by 5
  21. 2023Hyperbolic Geometry in Computer Vision: A Surveycited by 4
  22. 2024Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Contextcited by 4
  23. 2025Learning Along the Arrow of Time: Hyperbolic Geometry for Backward-Compatible Representation Learningcited by 4
  24. 2025HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Modelscited by 4
  25. 2024Hyperbolic Image-and-Pointcloud Contrastive Learning for 3D Classificationcited by 3
  26. 2024HypDAE: Hyperbolic Diffusion Autoencoders for Hierarchical Few-shot Image Generationcited by 3
  27. 2024Adversarial Attacks on Hyperbolic Networkscited by 3
  28. 2025Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?cited by 3
  29. 2022Bending the Future: Autoregressive Modeling of Temporal Knowledge Graphs in Curvature-Variable Hyperbolic Spacescited by 2
  30. 2022Spatio-Temporal Crop Aggregation for Video Representation Learningcited by 2
  31. 2025Multi-level and Multi-modal Action Anticipationcited by 2
  32. 2022Finding Islands of Predictability in Action Forecastingcited by 0
  33. 2023HLoOP -- Hyperbolic 2-space Local Outlier Probabilitiescited by 0
  34. 2025More Than A Shortcut: A Hyperbolic Approach To Early-Exit Networkscited by 0
  35. 2026Learning Action Hierarchies via Hybrid Geometric Diffusioncited by 0
  36. 2026Comparing Euclidean and Hyperbolic K-Means for Generalized Category Discoverycited by 0