# Multimodal Learning For Classroom Activity Detection

arXiv: https://arxiv.org/abs/1910.13799 · HTML: https://mlsanity.com/paper/1910.13799/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [Multi-Task Learning based Online Dialogic Instruction Detection with Pre-trained Language Models](https://mlsanity.com/paper/2107.07119/) (2021)
- [DialogID: A Dialogic Instruction Dataset for Improving Teaching Effectiveness in Online Environments](https://mlsanity.com/paper/2206.12034/) (2022)
- [Student Activity Recognition in Classroom Environments using Transfer Learning](https://mlsanity.com/paper/2312.00348/) (2023)

## Abstract (arXiv)

Classroom activity detection (CAD) focuses on accurately classifying whether the teacher or student is speaking and recording both the length of individual utterances during a class. A CAD solution helps teachers get instant feedback on their pedagogical instructions. This greatly improves educators' teaching skills and hence leads to students' achievement. However, CAD is very challenging because (1) the CAD model needs to be generalized well enough for different teachers and students; (2) data from both vocal and language modalities has to be wisely fused so that they can be complementary; and (3) the solution shouldn't heavily rely on additional recording device. In this paper, we address the above challenges by using a novel attention based neural framework. Our framework not only extracts both speech and language information, but utilizes attention mechanism to capture long-term semantic dependence. Our framework is device-free and is able to take any classroom recording as input. The proposed CAD learning framework is evaluated in two real-world education applications. The experimental results demonstrate the benefits of our approach on learning attention based neural network from classroom data with different modalities, and show our approach is able to outperform state-of-the-art baselines in terms of various evaluation metrics.