# PanoSent: A Panoptic Sextuple Extraction Benchmark for Multimodal Conversational Aspect-based Sentiment Analysis

arXiv: https://arxiv.org/abs/2408.09481 · HTML: https://mlsanity.com/paper/2408.09481/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 91 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Builds on (most cited first, 30 of 36)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023)
- [Knowing What, How and Why: A Near Complete Solution for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/1911.01616/) (2019)
- [WiCE: Real-World Entailment for Claims in Wikipedia](https://mlsanity.com/paper/2303.01432/) (2023)
- [Aspect Sentiment Quad Prediction as Paraphrase Generation](https://mlsanity.com/paper/2110.00796/) (2021)
- [A Unified Generative Framework for Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2106.04300/) (2021)
- [Reasoning Implicit Sentiment with Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.11255/) (2023)
- [Fine-Grained Semantically Aligned Vision-Language Pre-Training](https://mlsanity.com/paper/2208.02515/) (2022)
- [A Joint Training Dual-MRC Framework for Aspect Based Sentiment Analysis](https://mlsanity.com/paper/2101.00816/) (2021)
- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [MM-DFN: Multimodal Dynamic Fusion Network for Emotion Recognition in Conversations](https://mlsanity.com/paper/2203.02385/) (2022)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts](https://mlsanity.com/paper/2402.13220/) (2024)
- [Towards Semantic Equivalence of Tokenization in Multimodal LLM](https://mlsanity.com/paper/2406.05127/) (2024)
- [Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modeling](https://mlsanity.com/paper/1806.06228/) (2018)
- [Audio Retrieval with WavText5K and CLAP Training](https://mlsanity.com/paper/2209.14275/) (2022)
- [Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs](https://mlsanity.com/paper/2308.13812/) (2023)
- [Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment](https://mlsanity.com/paper/2406.19255/) (2024)
- [DiaASQ : A Benchmark of Conversational Aspect-based Sentiment Quadruple Analysis](https://mlsanity.com/paper/2211.05705/) (2022)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis](https://mlsanity.com/paper/2204.07955/) (2022)

## Built on by (most cited first, 13 of 13)

- [Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey](https://mlsanity.com/paper/2503.12605/) (2025)
- [Rethinking Driving World Model as Synthetic Data Generator for Perception Tasks](https://mlsanity.com/paper/2510.19195/) (2025)
- [Recent Advances in Multimodal Affective Computing: An NLP Perspective](https://mlsanity.com/paper/2409.07388/) (2024)
- [Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective](https://mlsanity.com/paper/2603.01083/) (2026)
- [OpenDeception: Learning Deception and Trust in Human-AI Interaction via Multi-Agent Simulation](https://mlsanity.com/paper/2504.13707/) (2025)
- [CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentrating](https://mlsanity.com/paper/2605.11723/) (2026)
- [Decoding the Flow: CauseMotion for Emotional Causality Analysis in Long-form Conversations](https://mlsanity.com/paper/2501.00778/) (2025)
- [Cross-lingual Aspect-Based Sentiment Analysis: A Survey on Tasks, Approaches, and Challenges](https://mlsanity.com/paper/2508.09516/) (2025)
- [Structured Prompting and LLM Ensembling for Multimodal Conversational Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2512.22603/) (2025)
- [Measuring the Unspoken: A Disentanglement Model and Benchmark for Psychological Analysis in the Wild](https://mlsanity.com/paper/2512.04728/) (2025)
- [Dialogue Summarization with Emotion Dynamics Using Topic- and Participant-Centric Decomposition](https://mlsanity.com/paper/2607.14769/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)
- [Omni-IO Skills: Harnessing Your Agent Omni-Native](https://mlsanity.com/paper/2609.31847/) (2026)

## Abstract (arXiv)

While existing Aspect-based Sentiment Analysis (ABSA) has received extensive effort and advancement, there are still gaps in defining a more holistic research target seamlessly integrating multimodality, conversation context, fine-granularity, and also covering the changing sentiment dynamics as well as cognitive causal rationales. This paper bridges the gaps by introducing a multimodal conversational ABSA, where two novel subtasks are proposed: 1) Panoptic Sentiment Sextuple Extraction, panoramically recognizing holder, target, aspect, opinion, sentiment, rationale from multi-turn multi-party multimodal dialogue. 2) Sentiment Flipping Analysis, detecting the dynamic sentiment transformation throughout the conversation with the causal reasons. To benchmark the tasks, we construct PanoSent, a dataset annotated both manually and automatically, featuring high quality, large scale, multimodality, multilingualism, multi-scenarios, and covering both implicit and explicit sentiment elements. To effectively address the tasks, we devise a novel Chain-of-Sentiment reasoning framework, together with a novel multimodal large language model (namely Sentica) and a paraphrase-based verification mechanism. Extensive evaluations demonstrate the superiority of our methods over strong baselines, validating the efficacy of all our proposed methods. The work is expected to open up a new era for the ABSA community, and thus all our codes and data are open at https://PanoSent.github.io/