# OpenD: A Benchmark for Language-Driven Door and Drawer Opening

arXiv: https://arxiv.org/abs/2212.05211 · HTML: https://mlsanity.com/paper/2212.05211/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 10 Dec 2022.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation](https://mlsanity.com/paper/2011.07215/) (2020)
- [ManipulaTHOR: A Framework for Visual Object Manipulation](https://mlsanity.com/paper/2104.11213/) (2021)
- [VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation](https://mlsanity.com/paper/2206.08522/) (2022)
- [DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following](https://mlsanity.com/paper/2202.13330/) (2022)
- [Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion](https://mlsanity.com/paper/2108.04927/) (2021)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)

## Built on by (most cited first, 3 of 3)

- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [Adaptive Mobile Manipulation for Articulated Objects In the Open World](https://mlsanity.com/paper/2401.14403/) (2024)
- [LoHoRavens: A Long-Horizon Language-Conditioned Benchmark for Robotic Tabletop Manipulation](https://mlsanity.com/paper/2310.12020/) (2023)

## Abstract (arXiv)

We introduce OPEND, a benchmark for learning how to use a hand to open cabinet doors or drawers in a photo-realistic and physics-reliable simulation environment driven by language instruction. To solve the task, we propose a multi-step planner composed of a deep neural network and rule-base controllers. The network is utilized to capture spatial relationships from images and understand semantic meaning from language instructions. Controllers efficiently execute the plan based on the spatial and semantic understanding. We evaluate our system by measuring its zero-shot performance in test data set. Experimental results demonstrate the effectiveness of decision planning by our multi-step planner for different hands, while suggesting that there is significant room for developing better models to address the challenge brought by language understanding, spatial reasoning, and long-term manipulation. We will release OPEND and host challenges to promote future research in this area.