# Using Large Multimodal Models to Extract Knowledge Components for Knowledge Tracing from Multimedia Question Information

arXiv: https://arxiv.org/abs/2409.20167 · HTML: https://mlsanity.com/paper/2409.20167/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 49 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 15 of 15)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [A Self-Attentive model for Knowledge Tracing](https://mlsanity.com/paper/1907.06837/) (2019)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing](https://mlsanity.com/paper/2002.07033/) (2020)
- [Knowledge Tracing: A Survey](https://mlsanity.com/paper/2201.06953/) (2022)
- [pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models](https://mlsanity.com/paper/2206.11460/) (2022)
- [DAS3H: Modeling Student Learning and Forgetting for Optimally Scheduling Distributed Practice of Skills](https://mlsanity.com/paper/1905.06873/) (2019)
- [Question Generation for Adaptive Education](https://mlsanity.com/paper/2106.04262/) (2021)
- [Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking](https://mlsanity.com/paper/2203.01552/) (2022)
- [Towards Modeling Learner Performance with Large Language Models](https://mlsanity.com/paper/2403.14661/) (2024)
- [Language Model Can Do Knowledge Tracing: Simple but Effective Method to Integrate Language Model and Knowledge Tracing Task](https://mlsanity.com/paper/2406.02893/) (2024)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Scaffolding Language Learning via Multi-modal Tutoring Systems with Pedagogical Instructions](https://mlsanity.com/paper/2404.03429/) (2024)
- [Evaluating the Knowledge Dependency of Questions](https://mlsanity.com/paper/2211.11902/) (2022)

## Built on by (most cited first, 2 of 2)

- [Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challenges](https://mlsanity.com/paper/2509.07946/) (2025)
- [Enhancing knowledge tracing robustness for new question cold start in Intelligent Tutoring Systems](https://mlsanity.com/paper/2512.07179/) (2025)

## Abstract (arXiv)

Knowledge tracing models have enabled a range of intelligent tutoring systems to provide feedback to students. However, existing methods for knowledge tracing in learning sciences are predominantly reliant on statistical data and instructor-defined knowledge components, making it challenging to integrate AI-generated educational content with traditional established methods. We propose a method for automatically extracting knowledge components from educational content using instruction-tuned large multimodal models. We validate this approach by comprehensively evaluating it against knowledge tracing benchmarks in five domains. Our results indicate that the automatically extracted knowledge components can effectively replace human-tagged labels, offering a promising direction for enhancing intelligent tutoring systems in limited-data scenarios, achieving more explainable assessments in educational settings, and laying the groundwork for automated assessment.