# Early Fusion for Goal Directed Robotic Vision

arXiv: https://arxiv.org/abs/1811.08824 · HTML: https://mlsanity.com/paper/1811.08824/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2018.

## Built on by (most cited first, 3 of 3)

- [Reinforcement Learning with Latent Flow](https://mlsanity.com/paper/2101.01857/) (2021)
- [Dual Goal Representations](https://mlsanity.com/paper/2510.06714/) (2025)
- [Deep Multimodal Fusion for Surgical Feedback Classification](https://mlsanity.com/paper/2312.03231/) (2023)

## Abstract (arXiv)

Building perceptual systems for robotics which perform well under tight computational budgets requires novel architectures which rethink the traditional computer vision pipeline. Modern vision architectures require the agent to build a summary representation of the entire scene, even if most of the input is irrelevant to the agent's current goal. In this work, we flip this paradigm, by introducing EarlyFusion vision models that condition on a goal to build custom representations for downstream tasks. We show that these goal specific representations can be learned more quickly, are substantially more parameter efficient, and more robust than existing attention mechanisms in our domain. We demonstrate the effectiveness of these methods on a simulated robotic item retrieval problem that is trained in a fully end-to-end manner via imitation learning.