# I Can Tell What I am Doing: Toward Real-World Natural Language Grounding of Robot Experiences

arXiv: https://arxiv.org/abs/2411.12960 · HTML: https://mlsanity.com/paper/2411.12960/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2024.

## Built on by (most cited first, 11 of 11)

- [Perception Tokens Enhance Visual Reasoning in Multimodal Language Models](https://mlsanity.com/paper/2412.03548/) (2024)
- [From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment](https://mlsanity.com/paper/2502.01828/) (2025)
- [Free-form language-based robotic reasoning and grasping](https://mlsanity.com/paper/2503.13082/) (2025)
- [RACAS: Controlling Diverse Robots With a Single Agentic System](https://mlsanity.com/paper/2603.05621/) (2026)
- [A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real World](https://mlsanity.com/paper/2507.16713/) (2025)
- [Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models](https://mlsanity.com/paper/2603.23149/) (2026)
- [KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis](https://mlsanity.com/paper/2604.07034/) (2026)
- [Personalised Explanations in Long-term Human-Robot Interactions](https://mlsanity.com/paper/2507.03049/) (2025)
- [Evaluating VLMs' Spatial Reasoning Over Robot Motion: A Step Towards Robot Planning with Motion Preferences](https://mlsanity.com/paper/2603.13100/) (2026)
- [Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deployment](https://mlsanity.com/paper/2604.11306/) (2026)
- [Controlling Collectives of AI Agents in Reasoning Space with Spatial Transformers](https://mlsanity.com/paper/2609.28247/) (2026)

## Abstract (arXiv)

Understanding robot behaviors and experiences through natural language is crucial for developing intelligent and transparent robotic systems. Recent advancement in large language models (LLMs) makes it possible to translate complex, multi-modal robotic experiences into coherent, human-readable narratives. However, grounding real-world robot experiences into natural language is challenging due to many reasons, such as multi-modal nature of data, differing sample rates, and data volume. We introduce RONAR, an LLM-based system that generates natural language narrations from robot experiences, aiding in behavior announcement, failure analysis, and human interaction to recover failure. Evaluated across various scenarios, RONAR outperforms state-of-the-art methods and improves failure recovery efficiency. Our contributions include a multi-modal framework for robot experience narration, a comprehensive real-robot dataset, and empirical evidence of RONAR's effectiveness in enhancing user experience in system transparency and failure analysis.