# iLearnRobot: An Interactive Learning-Based Multi-Modal Robot with Continuous Improvement

arXiv: https://arxiv.org/abs/2507.22896 · HTML: https://mlsanity.com/paper/2507.22896/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks](https://mlsanity.com/paper/1911.07883/) (2019)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections](https://mlsanity.com/paper/2311.10678/) (2023)
- [Improving Grounded Natural Language Understanding through Human-Robot Dialog](https://mlsanity.com/paper/1903.00122/) (2019)
- [MQA: Answering the Question via Robotic Manipulation](https://mlsanity.com/paper/2003.04641/) (2020)

## Abstract (arXiv)

It is crucial that robots' performance can be improved after deployment, as they are inherently likely to encounter novel scenarios never seen before. This paper presents an innovative solution: an interactive learning-based robot system powered by a Multi-modal Large Language Model(MLLM). A key feature of our system is its ability to learn from natural dialogues with non-expert users. We also propose chain of question to clarify the exact intent of the question before providing an answer and dual-modality retrieval modules to leverage these interaction events to avoid repeating same mistakes, ensuring a seamless user experience before model updates, which is in contrast to current mainstream MLLM-based robotic systems. Our system marks a novel approach in robotics by integrating interactive learning, paving the way for superior adaptability and performance in diverse environments. We demonstrate the effectiveness and improvement of our method through experiments, both quantitively and qualitatively.