# Building Knowledge from Interactions: An LLM-Based Architecture for Adaptive Tutoring and Social Reasoning

arXiv: https://arxiv.org/abs/2504.01588 · HTML: https://mlsanity.com/paper/2504.01588/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 8 of 8)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [Understanding Large-Language Model (LLM)-powered Human-Robot Interaction](https://mlsanity.com/paper/2401.03217/) (2024)
- [VITA: A Multi-modal LLM-based System for Longitudinal, Autonomous, and Adaptive Robotic Mental Well-being Coaching](https://mlsanity.com/paper/2312.09740/) (2023)
- [Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation](https://mlsanity.com/paper/2407.13505/) (2024)
- [A Roadmap for Embodied and Social Grounding in LLMs](https://mlsanity.com/paper/2409.16900/) (2024)

## Built on by (most cited first, 2 of 2)

- [Matching Game Preferences Through Dialogical Large Language Models: A Perspective](https://mlsanity.com/paper/2507.20000/) (2025)
- [ARIS: Agentic and Relationship Intelligence System for Social Robots](https://mlsanity.com/paper/2605.00943/) (2026)

## Abstract (arXiv)

Integrating robotics into everyday scenarios like tutoring or physical training requires robots capable of adaptive, socially engaging, and goal-oriented interactions. While Large Language Models show promise in human-like communication, their standalone use is hindered by memory constraints and contextual incoherence. This work presents a multimodal, cognitively inspired framework that enhances LLM-based autonomous decision-making in social and task-oriented Human-Robot Interaction. Specifically, we develop an LLM-based agent for a robot trainer, balancing social conversation with task guidance and goal-driven motivation. To further enhance autonomy and personalization, we introduce a memory system for selecting, storing and retrieving experiences, facilitating generalized reasoning based on knowledge built across different interactions. A preliminary HRI user study and offline experiments with a synthetic dataset validate our approach, demonstrating the system's ability to manage complex interactions, autonomously drive training tasks, and build and retrieve contextual memories, advancing socially intelligent robotics.