# Language and Sketching: An LLM-driven Interactive Multimodal Multitask Robot Navigation Framework

arXiv: https://arxiv.org/abs/2311.08244 · HTML: https://mlsanity.com/paper/2311.08244/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2023.

## Built on by (most cited first, 16 of 16)

- [Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions](https://mlsanity.com/paper/2408.04168/) (2024)
- [Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMs](https://mlsanity.com/paper/2408.06520/) (2024)
- [Multi-Agent LLM Actor-Critic Framework for Social Robot Navigation](https://mlsanity.com/paper/2503.09758/) (2025)
- [MARLIN: Multi-Agent Reinforcement Learning Guided by Language-Based Inter-Robot Negotiation](https://mlsanity.com/paper/2410.14383/) (2024)
- [Unifying Large Language Model and Deep Reinforcement Learning for Human-in-Loop Interactive Socially-aware Navigation](https://mlsanity.com/paper/2403.15648/) (2024)
- [Arena 4.0: A Comprehensive ROS2 Development and Benchmarking Platform for Human-centric Navigation Using Generative-Model-based Environment Generation](https://mlsanity.com/paper/2409.12471/) (2024)
- [MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitation](https://mlsanity.com/paper/2602.05048/) (2026)
- [CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot Interaction](https://mlsanity.com/paper/2410.01273/) (2024)
- [Boosting Team Modeling through Tempo-Relational Representation Learning](https://mlsanity.com/paper/2507.13305/) (2025)
- [An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction](https://mlsanity.com/paper/2602.20219/) (2026)
- [Interpretable Robot Control via Structured Behavior Trees and Large Language Models](https://mlsanity.com/paper/2508.09621/) (2025)
- [Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation](https://mlsanity.com/paper/2602.18941/) (2026)
- [Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2603.16558/) (2026)
- [Bridging Network Fragmentation: A Semantic-Augmented DRL Framework for UAV-aided VANETs](https://mlsanity.com/paper/2603.18871/) (2026)
- [Ablation Study of Multimodal Perception, Language Grounding, and Control for Human-Robot Interaction in an Object Detection and Grasping Task](https://mlsanity.com/paper/2605.00963/) (2026)
- [From Technical Metrics to User Perception: A User Study of a Multimodal Human-Robot Interaction System for Object Detection and Grasping](https://mlsanity.com/paper/2607.00530/) (2026)

## Abstract (arXiv)

The socially-aware navigation system has evolved to adeptly avoid various obstacles while performing multiple tasks, such as point-to-point navigation, human-following, and -guiding. However, a prominent gap persists: in Human-Robot Interaction (HRI), the procedure of communicating commands to robots demands intricate mathematical formulations. Furthermore, the transition between tasks does not quite possess the intuitive control and user-centric interactivity that one would desire. In this work, we propose an LLM-driven interactive multimodal multitask robot navigation framework, termed LIM2N, to solve the above new challenge in the navigation field. We achieve this by first introducing a multimodal interaction framework where language and hand-drawn inputs can serve as navigation constraints and control objectives. Next, a reinforcement learning agent is built to handle multiple tasks with the received information. Crucially, LIM2N creates smooth cooperation among the reasoning of multimodal input, multitask planning, and adaptation and processing of the intelligent sensing modules in the complicated system. Extensive experiments are conducted in both simulation and the real world demonstrating that LIM2N has superior user needs understanding, alongside an enhanced interactive experience.