mlsanityData to 7 Oct 2026

A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysis

arXiv 2006.15955 ↗First posted 29 Jun 2020cs.CL cs.HC cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

Understanding expressed sentiment and emotions are two crucial factors in human multimodal language. This paper describes a Transformer-based joint-encoding (TBJE) for the task of Emotion Recognition and Sentiment Analysis. In addition to use the Transformer architecture, our approach relies on a modular co-attention and a glimpse layer to jointly encode one or more modalities. The proposed solution has also been submitted to the ACL20: Second Grand-Challenge on Multimodal Language to be evaluated on the CMU-MOSEI dataset. The code to replicate the presented experiments is open-source: https://github.com/jbdel/MOSEI_UMONS.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 12; 2024: 7; 2025: 8; 2026: 3.

46 papers.

Sort:

By area:

  1. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  2. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  3. 2022MARLIN: Masked Autoencoder for facial video Representation LearnINgcited by 55
  4. 2022M2FNet: Multi-modal Fusion Network for Emotion Recognition in Conversationcited by 47
  5. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  6. 2022CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimationcited by 39
  7. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  8. 2022COGMEN: COntextualized GNN based Multimodal Emotion recognitioNcited by 29
  9. 2023Multimodal Multi-loss Fusion Network for Sentiment Analysiscited by 26
  10. 2023Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applicationscited by 20
  11. 2021A Review of Text Style Transfer using Deep Learningcited by 18
  12. 2023Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interactioncited by 18
34 more
  1. 2023Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Modelscited by 16
  2. 2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15
  3. 2023Joyful: Joint Modality Fusion and Graph Contrastive Learning for Multimodal Emotion Recognitioncited by 14
  4. 2022SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalitiescited by 11
  5. 2023CAD -- Contextual Multi-modal Alignment for Dynamic AVQAcited by 10
  6. 2021Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shiftscited by 9
  7. 2023Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregationcited by 8
  8. 2022Multimodal Emotion Recognition with Modality-Pairwise Unsupervised Contrastive Losscited by 7
  9. 2024Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysiscited by 7
  10. 2022MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysiscited by 6
  11. 2021Unsupervised Multimodal Language Representations using Convolutional Autoencoderscited by 4
  12. 2024End-to-end Semantic-centric Video-based Multimodal Affective Computingcited by 4
  13. 2022Multimodal Vision Transformers with Forced Attention for Behavior Analysiscited by 3
  14. 2022Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysiscited by 2
  15. 2024Infer Induced Sentiment of Comment Response to Video: A New Task, Dataset and Baselinecited by 2
  16. 2025MVP: Multimodal Emotion Recognition based on Video and Physiological Signalscited by 2
  17. 2025Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learningcited by 2
  18. 2021Leveraging Sentiment Analysis Knowledge to Solve Emotion Detection Taskscited by 1
  19. 2023Dynamic Causal Disentanglement Model for Dialogue Emotion Detectioncited by 1
  20. 2024Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognitioncited by 1
  21. 2025Structures Meet Semantics: Multimodal Fusion via Graph Contrastive Learningcited by 1
  22. 2022Multi-scale Cooperative Multimodal Transformers for Multimodal Sentiment Analysis in Videoscited by 0
  23. 2023Emotion Embeddings unicodex2014 Learning Stable and Homogeneous Abstractions from Heterogeneous Affective Datasetscited by 0
  24. 2023Deep Metric Loss for Multimodal Learningcited by 0
  25. 2024GCM-Net: Graph-enhanced Cross-Modal Infusion with a Metaheuristic-Driven Network for Video Sentiment and Emotion Analysiscited by 0
  26. 2024Push the Limit of Multi-modal Emotion Recognition by Prompting LLMs with Receptive-Field-Aware Attention Weightingcited by 0
  27. 2025CM3T: Framework for Efficient Multimodal Learning for Inhomogeneous Interaction Datasetscited by 0
  28. 2025Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognitioncited by 0
  29. 2025Conditional Information Bottleneck for Multimodal Fusion: Overcoming Shortcut Learning in Sarcasm Detectioncited by 0
  30. 2025Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversationscited by 0
  31. 2025A unified framework for detecting point and collective anomalies in operating system logs via collaborative transformerscited by 0
  32. 2026Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversationscited by 0
  33. 2026Leveraging Self-Paced Curriculum Learning for Enhanced Modality Balance in Multimodal Conversational Emotion Recognitioncited by 0
  34. 2026The Illusion of Balanced Multimodal Sentiment Analysis: Beyond the Limits of Optimization-Based Methodscited by 0