mlsanityData to 7 Oct 2026

Using Large Multimodal Models to Extract Knowledge Components for Knowledge Tracing from Multimedia Question Information

arXiv 2409.20167 ↗First posted 30 Sep 2024cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Knowledge tracing models have enabled a range of intelligent tutoring systems to provide feedback to students. However, existing methods for knowledge tracing in learning sciences are predominantly reliant on statistical data and instructor-defined knowledge components, making it challenging to integrate AI-generated educational content with traditional established methods. We propose a method for automatically extracting knowledge components from educational content using instruction-tuned large multimodal models. We validate this approach by comprehensively evaluating it against knowledge tracing benchmarks in five domains. Our results indicate that the automatically extracted knowledge components can effectively replace human-tagged labels, offering a promising direction for enhancing intelligent tutoring systems in limited-data scenarios, achieving more explainable assessments in educational settings, and laying the groundwork for automated assessment.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 49 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

15 of 49 references.

References · 15

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  3. 2019A Self-Attentive model for Knowledge Tracingcited by 169
  4. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  5. 2020Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracingcited by 79
  6. 2022Knowledge Tracing: A Surveycited by 76
  7. 2022pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Modelscited by 50
  8. 2019DAS3H: Modeling Student Learning and Forgetting for Optimally Scheduling Distributed Practice of Skillscited by 26
  9. 2021Question Generation for Adaptive Educationcited by 22
  10. 2022Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Trackingcited by 21
  11. 2024Towards Modeling Learner Performance with Large Language Modelscited by 20
  12. 2024Language Model Can Do Knowledge Tracing: Simple but Effective Method to Integrate Language Model and Knowledge Tracing Taskcited by 20
3 more
  1. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  2. 2024Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructionscited by 11
  3. 2022Evaluating the Knowledge Dependency of Questionscited by 9

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challengescited by 0
  2. 2025Enhancing knowledge tracing robustness for new question cold start in Intelligent Tutoring Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards LLM-Empowered Knowledge Tracing via LLM-Student Hierarchical Behavior Alignment in Hyperbolic Space
  2. 2023How to Build an Adaptive AI Tutor for Any Course Using Knowledge Graph-Enhanced Retrieval-Augmented Generation (KG-RAG)
  3. 2024Leveraging Large Language Models to Generate Course-specific Semantically Annotated Learning Objects
  4. 2025LLM-driven Effective Knowledge Tracing by Integrating Dual-channel Difficulty
  5. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
  6. 2025Evaluating Multimodal Large Language Models on Educational Textbook Question Answering
  7. 2024Integrating Cognitive AI with Generative Models for Enhanced Question Answering in Skill-based Learning
  8. 2026Evidence-Grounded Multimodal Knowledge Graph Construction for Multi-Lecture Educational Reasoning