# CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction

arXiv: https://arxiv.org/abs/2410.01273 · HTML: https://mlsanity.com/paper/2410.01273/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 38 of its 57 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 30 of 38)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Visual Language Maps for Robot Navigation](https://mlsanity.com/paper/2210.05714/) (2022)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation](https://mlsanity.com/paper/2406.06978/) (2024)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation](https://mlsanity.com/paper/2402.15852/) (2024)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Behavior Generation with Latent Actions](https://mlsanity.com/paper/2403.03181/) (2024)
- [A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges](https://mlsanity.com/paper/2309.02473/) (2023)
- [RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches](https://mlsanity.com/paper/2311.01977/) (2023)
- [Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions](https://mlsanity.com/paper/2309.10150/) (2023)
- [ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation](https://mlsanity.com/paper/2301.13166/) (2023)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [Deep Visual MPC-Policy Learning for Navigation](https://mlsanity.com/paper/1903.02749/) (2019)
- [Discrete Sequential Prediction of Continuous Actions for Deep RL](https://mlsanity.com/paper/1705.05035/) (2017)
- [Vision-Language Navigation: A Survey and Taxonomy](https://mlsanity.com/paper/2108.11544/) (2021)
- [VOILA: Visual-Observation-Only Imitation Learning for Autonomous Navigation](https://mlsanity.com/paper/2105.09371/) (2021)
- [Continuous Control with Action Quantization from Demonstrations](https://mlsanity.com/paper/2110.10149/) (2021)
- [Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework](https://mlsanity.com/paper/2311.08244/) (2023)

## Built on by (most cited first, 3 of 3)

- [D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI](https://mlsanity.com/paper/2510.05684/) (2025)
- [CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agents](https://mlsanity.com/paper/2511.20216/) (2025)
- [Commonsense-Grounded Path Planning from Abstract Instructions](https://mlsanity.com/paper/2609.22813/) (2026)

## Abstract (arXiv)

Real-life robot navigation involves more than just reaching a destination; it requires optimizing movements while addressing scenario-specific goals. An intuitive way for humans to express these goals is through abstract cues like verbal commands or rough sketches. Such human guidance may lack details or be noisy. Nonetheless, we expect robots to navigate as intended. For robots to interpret and execute these abstract instructions in line with human expectations, they must share a common understanding of basic navigation concepts with humans. To this end, we introduce CANVAS, a novel framework that combines visual and linguistic instructions for commonsense-aware navigation. Its success is driven by imitation learning, enabling the robot to learn from human navigation behavior. We present COMMAND, a comprehensive dataset with human-annotated navigation results, spanning over 48 hours and 219 km, designed to train commonsense-aware navigation systems in simulated environments. Our experiments show that CANVAS outperforms the strong rule-based system ROS NavStack across all environments, demonstrating superior performance with noisy instructions. Notably, in the orchard environment, where ROS NavStack records a 0% total success rate, CANVAS achieves a total success rate of 67%. CANVAS also closely aligns with human demonstrations and commonsense constraints, even in unseen environments. Furthermore, real-world deployment of CANVAS showcases impressive Sim2Real transfer with a total success rate of 69%, highlighting the potential of learning from human demonstrations in simulated environments for real-world applications.