# Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks

arXiv: https://arxiv.org/abs/2602.13588 · HTML: https://mlsanity.com/paper/2602.13588/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 47 of its 62 references that are in the map.
- First posted on arXiv 14 Feb 2026.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Virtual KITTI 2](https://mlsanity.com/paper/2001.10773/) (2020)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [K-Net: Towards Unified Image Segmentation](https://mlsanity.com/paper/2106.14855/) (2021)
- [Iterative Geometry Encoding Volume for Stereo Matching](https://mlsanity.com/paper/2303.06615/) (2023)
- [Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance](https://mlsanity.com/paper/2007.06936/) (2020)
- [SelFlow: Self-Supervised Learning of Optical Flow](https://mlsanity.com/paper/1904.09117/) (2019)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [SegStereo: Exploiting Semantic Information for Disparity Estimation](https://mlsanity.com/paper/1807.11699/) (2018)
- [Accurate Optical Flow via Direct Cost Volume Processing](https://mlsanity.com/paper/1704.07325/) (2017)
- [SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection](https://mlsanity.com/paper/2008.11351/) (2020)
- [ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions](https://mlsanity.com/paper/2403.07392/) (2024)
- [Sparse Instance Activation for Real-Time Instance Segmentation](https://mlsanity.com/paper/2203.12827/) (2022)
- [DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation](https://mlsanity.com/paper/2309.09668/) (2023)

## Abstract (arXiv)

Inspired by the human visual system, which operates on two parallel yet interactive streams for contextual and spatial understanding, this article presents Two Interactive Streams (TwInS), a novel bio-inspired joint learning framework capable of simultaneously performing scene parsing and geometric vision tasks. TwInS adopts a unified, general-purpose architecture in which multi-level contextual features from the scene parsing stream are infused into the geometric vision stream to guide its iterative refinement. In the reverse direction, decoded geometric features are projected into the contextual feature space for selective heterogeneous feature fusion via a novel cross-task adapter, which leverages rich cross-view geometric cues to enhance scene parsing. To eliminate the dependence on costly human-annotated correspondence ground truth, TwInS is further equipped with a tailored semi-supervised training strategy, which unleashes the potential of large-scale multi-view data and enables continuous self-evolution without requiring ground-truth correspondences. Extensive experiments conducted on three public datasets validate the effectiveness of TwInS's core components and demonstrate its superior performance over existing state-of-the-art approaches. The source code will be made publicly available upon publication.