# Visionary: Vision architecture discovery for robot learning

arXiv: https://arxiv.org/abs/2103.14633 · HTML: https://mlsanity.com/paper/2103.14633/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 26 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours](https://mlsanity.com/paper/1509.06825/) (2015)
- [Hierarchical Representations for Efficient Architecture Search](https://mlsanity.com/paper/1711.00436/) (2017)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [Zero-Shot Visual Imitation](https://mlsanity.com/paper/1804.08606/) (2018)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Evolving Deep Neural Networks](https://mlsanity.com/paper/1703.00548/) (2017)
- [Learning to Manipulate Deformable Objects without Demonstrations](https://mlsanity.com/paper/1910.13439/) (2019)
- [AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures](https://mlsanity.com/paper/1905.13209/) (2019)
- [The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation](https://mlsanity.com/paper/1907.13236/) (2019)
- [High precision grasp pose detection in dense clutter](https://mlsanity.com/paper/1603.01564/) (2016)
- [Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods](https://mlsanity.com/paper/1802.10264/) (2018)
- [Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter](https://mlsanity.com/paper/1903.01588/) (2019)
- [Learning a visuomotor controller for real world robotic grasping using simulated depth images](https://mlsanity.com/paper/1706.04652/) (2017)
- [AssembleNet++: Assembling Modality Representations via Attention Connections](https://mlsanity.com/paper/2008.08072/) (2020)
- [Tiny Video Networks](https://mlsanity.com/paper/1910.06961/) (2019)
- [Learning Precise 3D Manipulation from Multiple Uncalibrated Cameras](https://mlsanity.com/paper/2002.09107/) (2020)
- [Evolving Space-Time Neural Architectures for Videos](https://mlsanity.com/paper/1811.10636/) (2018)
- [TextNAS: A Neural Architecture Search Space tailored for Text Representation](https://mlsanity.com/paper/1912.10729/) (2019)
- [An Introduction to Neural Architecture Search for Convolutional Networks](https://mlsanity.com/paper/2005.11074/) (2020)

## Built on by (most cited first, 4 of 4)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?](https://mlsanity.com/paper/2206.05266/) (2022)
- [Differentiable Architecture Search for Reinforcement Learning](https://mlsanity.com/paper/2106.02229/) (2021)
- [HyperPPO: A scalable method for finding small policies for robotic control](https://mlsanity.com/paper/2309.16663/) (2023)

## Abstract (arXiv)

We propose a vision-based architecture search algorithm for robot manipulation learning, which discovers interactions between low dimension action inputs and high dimensional visual inputs. Our approach automatically designs architectures while training on the task - discovering novel ways of combining and attending image feature representations with actions as well as features from previous layers. The obtained new architectures demonstrate better task success rates, in some cases with a large margin, compared to a recent high performing baseline. Our real robot experiments also confirm that it improves grasping performance by 6%. This is the first approach to demonstrate a successful neural architecture search and attention connectivity search for a real-robot task.