# ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectives

arXiv: https://arxiv.org/abs/2411.19083 · HTML: https://mlsanity.com/paper/2411.19083/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 76 references that are in the map.
- First posted on arXiv 28 Nov 2024.

## Builds on (most cited first, 30 of 60)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model](https://mlsanity.com/paper/2207.07115/) (2022)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [H2O: Two Hands Manipulating Objects for First Person Interaction Recognition](https://mlsanity.com/paper/2104.11181/) (2021)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [PixelLM: Pixel Reasoning with Large Multimodal Model](https://mlsanity.com/paper/2312.02228/) (2023)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Universal Instance Perception as Object Discovery and Retrieval](https://mlsanity.com/paper/2303.06674/) (2023)
- [HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World](https://mlsanity.com/paper/2309.17024/) (2023)

## Built on by (most cited first, 29 of 29)

- [Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods](https://mlsanity.com/paper/2510.07143/) (2025)
- [Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis](https://mlsanity.com/paper/2511.20186/) (2025)
- [CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning](https://mlsanity.com/paper/2506.17629/) (2025)
- [StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios](https://mlsanity.com/paper/2512.04451/) (2025)
- [EgoSound: Benchmarking Sound Understanding in Egocentric Videos](https://mlsanity.com/paper/2602.14122/) (2026)
- [VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning](https://mlsanity.com/paper/2603.01195/) (2026)
- [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026)
- [Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting](https://mlsanity.com/paper/2512.20014/) (2025)
- [Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation](https://mlsanity.com/paper/2606.00095/) (2026)
- [Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence](https://mlsanity.com/paper/2607.12477/) (2026)
- [Cross-View Multi-Modal Segmentation @ Ego-Exo4D Challenges 2025](https://mlsanity.com/paper/2506.05856/) (2025)
- [ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding](https://mlsanity.com/paper/2607.28312/) (2026)
- [VGGT-Segmentor: Geometry-Enhanced Cross-View Segmentation](https://mlsanity.com/paper/2604.13596/) (2026)
- [CrossView Suite: Harnessing Cross-view Spatial Intelligence of MLLMs with Dataset, Model and Benchmark](https://mlsanity.com/paper/2605.18621/) (2026)
- [The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models](https://mlsanity.com/paper/2601.08876/) (2026)
- [IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory](https://mlsanity.com/paper/2604.20136/) (2026)
- [ReefNet: A Large-Scale Dataset and Benchmark for Fine-Grained Coral Reef Recognition](https://mlsanity.com/paper/2510.16822/) (2025)
- [EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next](https://mlsanity.com/paper/2603.12147/) (2026)
- [From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation](https://mlsanity.com/paper/2604.13793/) (2026)
- [Once-For-All: A Train-Once and Select-Anytime Framework for Multimodal Instruction Tuning](https://mlsanity.com/paper/2605.26761/) (2026)
- [Sharpness-aware Model Merging with Salience Recovery for LLM-based Cross-Domain Sequential Recommendation](https://mlsanity.com/paper/2607.25366/) (2026)
- [MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understanding](https://mlsanity.com/paper/2609.38900/) (2026)
- [Self-supervised pretraining for an iterative image size agnostic vision transformer](https://mlsanity.com/paper/2604.20392/) (2026)
- [InfoGeo: Information-Theoretic Object-Centric Learning for Cross-View Generalizable UAV Geo-Localization](https://mlsanity.com/paper/2605.07099/) (2026)
- [The First EgoCross Challenge at EgoVis 2026: Cross-Domain Egocentric Video Question Answering](https://mlsanity.com/paper/2608.04589/) (2026)
- [Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation](https://mlsanity.com/paper/2609.35368/) (2026)
- [Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation](https://mlsanity.com/paper/2610.12783/) (2026)
- [EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos](https://mlsanity.com/paper/2610.17128/) (2026)
- [EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions](https://mlsanity.com/paper/2610.18499/) (2026)

## Abstract (arXiv)

Bridging the gap between ego-centric and exo-centric views has been a long-standing question in computer vision. In this paper, we focus on the emerging Ego-Exo object correspondence task, which aims to understand object relations across ego-exo perspectives through segmentation. While numerous segmentation models have been proposed, most operate on a single image (view), making them impractical for cross-view scenarios. PSALM, a recently proposed segmentation method, stands out as a notable exception with its demonstrated zero-shot ability on this task. However, due to the drastic viewpoint change between ego and exo, PSALM fails to accurately locate and segment objects, especially in complex backgrounds or when object appearances change significantly. To address these issues, we propose ObjectRelator, a novel approach featuring two key modules: Multimodal Condition Fusion (MCFuse) and SSL-based Cross-View Object Alignment (XObjAlign). MCFuse introduces language as an additional cue, integrating both visual masks and textual descriptions to improve object localization and prevent incorrect associations. XObjAlign enforces cross-view consistency through self-supervised alignment, enhancing robustness to object appearance variations. Extensive experiments demonstrate ObjectRelator's effectiveness on the large-scale Ego-Exo4D benchmark and HANDAL-X (an adapted dataset for cross-view segmentation) with state-of-the-art performance. Code is made available at: http://yuqianfu.com/ObjectRelator.