# Self-supervised Hypergraphs for Learning Multiple World Interpretations

arXiv: https://arxiv.org/abs/2308.07615 · HTML: https://mlsanity.com/paper/2308.07615/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 15 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Hypergraph Neural Networks](https://mlsanity.com/paper/1809.09401/) (2018)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction](https://mlsanity.com/paper/1611.09842/) (2016)
- [Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras](https://mlsanity.com/paper/1904.04998/) (2019)
- [Semantically-Guided Representation Learning for Self-Supervised Monocular Depth](https://mlsanity.com/paper/2002.12319/) (2020)
- [Self-supervised Learning with Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera](https://mlsanity.com/paper/1907.05820/) (2019)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation](https://mlsanity.com/paper/2204.05525/) (2022)
- [FasterSeg: Searching for Faster Real-time Semantic Segmentation](https://mlsanity.com/paper/1912.10917/) (2019)
- [Auto-Lambda: Disentangling Dynamic Task Relationships](https://mlsanity.com/paper/2202.03091/) (2022)
- [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019)
- [Joint Semantic Segmentation and Boundary Detection using Iterative Pyramid Contexts](https://mlsanity.com/paper/2004.07684/) (2020)
- [Distilled Semantics for Comprehensive Scene Understanding from Videos](https://mlsanity.com/paper/2003.14030/) (2020)
- [Robustness via Cross-Domain Ensembles](https://mlsanity.com/paper/2103.10919/) (2021)
- [FBSNet: A Fast Bilateral Symmetrical Network for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2109.00699/) (2021)
- [Semantics through Time: Semi-supervised Segmentation of Aerial Videos with Iterative Label Propagation](https://mlsanity.com/paper/2010.01910/) (2020)
- [Consensus Clustering With Unsupervised Representation Learning](https://mlsanity.com/paper/2010.01245/) (2020)
- [X-Distill: Improving Self-Supervised Monocular Depth via Cross-Task Distillation](https://mlsanity.com/paper/2110.12516/) (2021)
- [A nonlinear diffusion method for semi-supervised learning on hypergraphs](https://mlsanity.com/paper/2103.14867/) (2021)

## Built on by (most cited first, 8 of 8)

- [OccuFly: A 3D Vision Benchmark for Semantic Scene Completion from the Aerial Perspective](https://mlsanity.com/paper/2512.20770/) (2025)
- [ClaraVid: A Holistic Scene Reconstruction Benchmark From Aerial Perspective With Delentropy-Based Complexity Profiling](https://mlsanity.com/paper/2503.17856/) (2025)
- [Learning from Random Subspace Exploration: Generalized Test-Time Augmentation with Self-supervised Distillation](https://mlsanity.com/paper/2507.01347/) (2025)
- [Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning](https://mlsanity.com/paper/2510.10068/) (2025)
- [Deep Temporal Graph Clustering: A Comprehensive Benchmark and Datasets](https://mlsanity.com/paper/2601.12903/) (2026)
- [Multiple Random Masking Autoencoder Ensembles for Robust Multimodal Semi-supervised Learning](https://mlsanity.com/paper/2402.08035/) (2024)
- [Multi-modal video data-pipelines for machine learning with minimal human supervision](https://mlsanity.com/paper/2510.14862/) (2025)
- [Self-Supervised Learning to Fly using Efficient Semantic Segmentation and Metric Depth Estimation for Low-Cost Autonomous UAVs](https://mlsanity.com/paper/2510.16624/) (2025)

## Abstract (arXiv)

We present a method for learning multiple scene representations given a small labeled set, by exploiting the relationships between such representations in the form of a multi-task hypergraph. We also show how we can use the hypergraph to improve a powerful pretrained VisTransformer model without any additional labeled data. In our hypergraph, each node is an interpretation layer (e.g., depth or segmentation) of the scene. Within each hyperedge, one or several input nodes predict the layer at the output node. Thus, each node could be an input node in some hyperedges and an output node in others. In this way, multiple paths can reach the same node, to form ensembles from which we obtain robust pseudolabels, which allow self-supervised learning in the hypergraph. We test different ensemble models and different types of hyperedges and show superior performance to other multi-task graph models in the field. We also introduce Dronescapes, a large video dataset captured with UAVs in different complex real-world scenes, with multiple representations, suitable for multi-task learning.