# A Multimodal Fusion Network For Student Emotion Recognition Based on Transformer and Tensor Product

arXiv: https://arxiv.org/abs/2403.08511 · HTML: https://mlsanity.com/paper/2403.08511/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 13 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Builds on (most cited first, 7 of 7)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Multimodal Intelligence: Representation Learning, Information Fusion, and Applications](https://mlsanity.com/paper/1911.03977/) (2019)
- [An Integrative Paradigm for Enhanced Stroke Prediction: Synergizing XGBoost and xDeepFM Algorithms](https://mlsanity.com/paper/2310.16430/) (2023)

## Built on by (most cited first, 22 of 22)

- [Optimizing Automated Picking Systems in Warehouse Robots Using Machine Learning](https://mlsanity.com/paper/2408.16633/) (2024)
- [A Neural Matrix Decomposition Recommender System Model based on the Multimodal Large Language Model](https://mlsanity.com/paper/2407.08942/) (2024)
- [Transformer-Based Classification Outcome Prediction for Multimodal Stroke Treatment](https://mlsanity.com/paper/2404.12634/) (2024)
- [Research on the Design of a Short Video Recommendation System Based on Multimodal Information and Differential Privacy](https://mlsanity.com/paper/2504.08751/) (2025)
- [User Behavior Analysis in Privacy Protection with Large Language Models: A Study on Privacy Preferences with Limited Data](https://mlsanity.com/paper/2505.06305/) (2025)
- [Research on Personalized Financial Product Recommendation by Integrating Large Language Models and Graph Neural Networks](https://mlsanity.com/paper/2506.05873/) (2025)
- [Research on Model Parallelism and Data Parallelism Optimization Methods in Large Language Model-Based Recommendation Systems](https://mlsanity.com/paper/2506.17551/) (2025)
- [Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysis](https://mlsanity.com/paper/2602.16144/) (2026)
- [Meta-Learning for Cold-Start Personalization in Prompt-Tuned LLMs](https://mlsanity.com/paper/2507.16672/) (2025)
- [RLHF Fine-Tuning of LLMs for Alignment with Implicit User Feedback in Conversational Recommenders](https://mlsanity.com/paper/2508.05289/) (2025)
- [Analysis and Design of a Personalized Recommendation System Based on a Dynamic User Interest Model](https://mlsanity.com/paper/2410.09923/) (2024)
- [Real-time Video Target Tracking Algorithm Utilizing Convolutional Neural Networks (CNN)](https://mlsanity.com/paper/2411.18314/) (2024)
- [Deep Learning Model Acceleration and Optimization Strategies for Real-Time Recommendation Systems](https://mlsanity.com/paper/2506.11421/) (2025)
- [Detecting and Classifying Defective Products in Images Using YOLO](https://mlsanity.com/paper/2412.16935/) (2024)
- [LLM-Augmented Symptom Analysis for Cardiovascular Disease Risk Prediction: A Clinical NLP](https://mlsanity.com/paper/2507.11052/) (2025)
- [Efficient Cold-Start Recommendation via BPE Token-Level Embedding Initialization with LLM](https://mlsanity.com/paper/2509.13179/) (2025)
- [Multi-modal clothing recommendation model based on large model and VAE enhancement](https://mlsanity.com/paper/2410.02219/) (2024)
- [Optimization and Scalability of Collaborative Filtering Algorithms in Large Language Models](https://mlsanity.com/paper/2412.18715/) (2024)
- [The Transparent Earth: A Multimodal Foundation Model for the Earth's Subsurface](https://mlsanity.com/paper/2509.02783/) (2025)
- [Instructional Prompt Optimization for Few-Shot LLM-Based Recommendations on Cold-Start Users](https://mlsanity.com/paper/2509.09066/) (2025)
- [Using Vision Language Models to Detect Students' Academic Emotion through Facial Expressions](https://mlsanity.com/paper/2506.10334/) (2025)
- [Multimodal Foundation Model-Driven User Interest Modeling and Behavior Analysis on Short Video Platforms](https://mlsanity.com/paper/2509.04751/) (2025)

## Abstract (arXiv)

This paper introduces a new multi-modal model based on the Transformer architecture and tensor product fusion strategy, combining BERT's text vectors and ViT's image vectors to classify students' psychological conditions, with an accuracy of 93.65%. The purpose of the study is to accurately analyze the mental health status of students from various data sources. This paper discusses modal fusion methods, including early, late and intermediate fusion, to overcome the challenges of integrating multi-modal information. Ablation studies compare the performance of different models and fusion techniques, showing that the proposed model outperforms existing methods such as CLIP and ViLBERT in terms of accuracy and inference speed. Conclusions indicate that while this model has significant advantages in emotion recognition, its potential to incorporate other data modalities provides areas for future research.