# Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions

arXiv: https://arxiv.org/abs/2404.03429 · HTML: https://mlsanity.com/paper/2404.03429/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 63 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 11 of 11)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education](https://mlsanity.com/paper/2308.02773/) (2023)
- [Evaluating Spatial Understanding of Large Language Models](https://mlsanity.com/paper/2310.14540/) (2023)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [Opportunities and Challenges in Neural Dialog Tutoring](https://mlsanity.com/paper/2301.09919/) (2023)
- [Strategize Before Teaching: A Conversational Tutoring System with Pedagogy Self-Distillation](https://mlsanity.com/paper/2302.13496/) (2023)
- [Empowering Private Tutoring by Chaining Large Language Models](https://mlsanity.com/paper/2309.08112/) (2023)

## Built on by (most cited first, 12 of 12)

- [Training Turn-by-Turn Verifiers for Dialogue Tutoring Agents: The Curious Case of LLMs as Your Coding Tutors](https://mlsanity.com/paper/2502.13311/) (2025)
- [EduPlanner: LLM-Based Multi-Agent Systems for Customized and Intelligent Instructional Design](https://mlsanity.com/paper/2504.05370/) (2025)
- [Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems](https://mlsanity.com/paper/2404.06762/) (2024)
- [AI-Powered Math Tutoring: Platform for Personalized and Adaptive Education](https://mlsanity.com/paper/2507.12484/) (2025)
- [Awaking the Slides: A Tuning-free and Knowledge-regulated AI Tutoring System via Language Model Coordination](https://mlsanity.com/paper/2409.07372/) (2024)
- [Using Large Multimodal Models to Extract Knowledge Components for Knowledge Tracing from Multimedia Question Information](https://mlsanity.com/paper/2409.20167/) (2024)
- [SingaKids: A Multilingual Multimodal Dialogic Tutor for Language Learning](https://mlsanity.com/paper/2506.02412/) (2025)
- [COGENT: A Curriculum-oriented Framework for Generating Grade-appropriate Educational Content](https://mlsanity.com/paper/2506.09367/) (2025)
- [A Hierarchical Probabilistic Framework for Incremental Knowledge Tracing in Classroom Settings](https://mlsanity.com/paper/2506.09393/) (2025)
- [The Imperfect Learner: Incorporating Developmental Trajectories in Memory-based Student Simulation](https://mlsanity.com/paper/2511.05903/) (2025)
- [Robust Multilingual Text-to-Pictogram Mapping for Scalable Reading Rehabilitation](https://mlsanity.com/paper/2603.24536/) (2026)
- [Do LLM Tutors Actually Teach? A Process-Centered Synthesis of the Tutoring Loop](https://mlsanity.com/paper/2610.15692/) (2026)

## Abstract (arXiv)

Intelligent tutoring systems (ITSs) that imitate human tutors and aim to provide immediate and customized instructions or feedback to learners have shown their effectiveness in education. With the emergence of generative artificial intelligence, large language models (LLMs) further entitle the systems to complex and coherent conversational interactions. These systems would be of great help in language education as it involves developing skills in communication, which, however, drew relatively less attention. Additionally, due to the complicated cognitive development at younger ages, more endeavors are needed for practical uses. Scaffolding refers to a teaching technique where teachers provide support and guidance to students for learning and developing new concepts or skills. It is an effective way to support diverse learning needs, goals, processes, and outcomes. In this work, we investigate how pedagogical instructions facilitate the scaffolding in ITSs, by conducting a case study on guiding children to describe images for language learning. We construct different types of scaffolding tutoring systems grounded in four fundamental learning theories: knowledge construction, inquiry-based learning, dialogic teaching, and zone of proximal development. For qualitative and quantitative analyses, we build and refine a seven-dimension rubric to evaluate the scaffolding process. In our experiment on GPT-4V, we observe that LLMs demonstrate strong potential to follow pedagogical instructions and achieve self-paced learning in different student groups. Moreover, we extend our evaluation framework from a manual to an automated approach, paving the way to benchmark various conversational tutoring systems.