# Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

arXiv: https://arxiv.org/abs/2408.14491 · HTML: https://mlsanity.com/paper/2408.14491/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 154 references that are in the map.
- First posted on arXiv 22 Aug 2024.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Hierarchical Video-Moment Retrieval and Step-Captioning](https://mlsanity.com/paper/2303.16406/) (2023)
- [Explainable Artificial Intelligence (XAI) on TimeSeries Data: A Survey](https://mlsanity.com/paper/2104.00950/) (2021)
- [A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science](https://mlsanity.com/paper/2403.14565/) (2024)
- [DiagrammerGPT: Generating Open-Domain, Open-Platform Diagrams via LLM Planning](https://mlsanity.com/paper/2310.12128/) (2023)
- [Multimodal Engagement Analysis from Facial Videos in the Classroom](https://mlsanity.com/paper/2101.04215/) (2021)
- [Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis](https://mlsanity.com/paper/2405.03677/) (2024)
- [Improving prediction of students' performance in intelligent tutoring systems using attribute selection and ensembles of different multimodal data sources](https://mlsanity.com/paper/2403.07194/) (2024)
- [A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments](https://mlsanity.com/paper/2405.06203/) (2024)
- [Unobtrusive and Multimodal Approach for Behavioral Engagement Detection of Students](https://mlsanity.com/paper/1901.05835/) (2019)
- [Multi-source and multimodal data fusion for predicting academic performance in blended learning university courses](https://mlsanity.com/paper/2403.05552/) (2024)

## Built on by (most cited first, 4 of 4)

- [LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports](https://mlsanity.com/paper/2503.02099/) (2025)
- [Multimodal Representation Learning and Fusion](https://mlsanity.com/paper/2506.20494/) (2025)
- [AI Misuse in Education Is a Measurement Problem: Toward a Learning Visibility Framework](https://mlsanity.com/paper/2603.07834/) (2026)
- [A Theory-Guided LLM Pedagogical Agent for STEM+C Scaffolding Without Over-Reliance](https://mlsanity.com/paper/2605.30539/) (2026)

## Abstract (arXiv)

Recent technological advancements in multimodal machine learning--including the rise of large language models (LLMs)--have improved our ability to collect, process, and analyze diverse multimodal data such as speech, video, and eye gaze in learning and training contexts. While prior reviews have addressed individual components of the multimodal pipeline (e.g., conceptual models, data fusion), a comprehensive review of empirical methods in applied multimodal environments remains notably absent. This review addresses that, introducing a taxonomy and framework that capture both established practices and recent innovations driven by LLMs and generative AI. We identify five modality groups: Natural Language, Vision, Physiological Signals, Human-Centered Evidence, and Environment Logs. Our analysis reveals that integrating modalities enables richer insights into learner and trainee behaviors, revealing latent patterns often overlooked by unimodal approaches. However, persistent challenges in multimodal data collection and integration continue to hinder the adoption of these systems in real-time classroom settings.