# Omni-SILA: Towards Omni-scene Driven Visual Sentiment Identifying, Locating and Attributing in Videos

arXiv: https://arxiv.org/abs/2503.00049 · HTML: https://mlsanity.com/paper/2503.00049/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 72 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [PandaGPT: One Model To Instruction-Follow Them All](https://mlsanity.com/paper/2305.16355/) (2023)
- [From Sparse to Soft Mixtures of Experts](https://mlsanity.com/paper/2308.00951/) (2023)
- [Valley: Video Assistant with Large Language model Enhanced abilitY](https://mlsanity.com/paper/2306.07207/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens](https://mlsanity.com/paper/2404.03413/) (2024)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [HawkEye: Training Video-Text LLMs for Grounding Text in Videos](https://mlsanity.com/paper/2403.10228/) (2024)
- [EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes](https://mlsanity.com/paper/2307.07961/) (2023)
- [CoLA: Weakly-Supervised Temporal Action Localization with Snippet Contrastive Learning](https://mlsanity.com/paper/2103.16392/) (2021)
- [Holmes-VAD: Towards Unbiased and Explainable Video Anomaly Detection via Multi-modal LLM](https://mlsanity.com/paper/2406.12235/) (2024)
- [UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition](https://mlsanity.com/paper/2211.11256/) (2022)
- [Uncovering What, Why and How: A Comprehensive Benchmark for Causation Understanding of Video Anomaly](https://mlsanity.com/paper/2405.00181/) (2024)
- [EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning](https://mlsanity.com/paper/2404.16670/) (2024)
- [GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation](https://mlsanity.com/paper/2311.16511/) (2023)
- [3DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding](https://mlsanity.com/paper/2401.03201/) (2024)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Stimuli-Aware Visual Emotion Analysis](https://mlsanity.com/paper/2109.01812/) (2021)
- [PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression](https://mlsanity.com/paper/1909.05693/) (2019)

## Built on by (most cited first, 1 of 1)

- [AffectSeek: Agentic Affective Understanding in Long Videos under Vague User Queries](https://mlsanity.com/paper/2605.05640/) (2026)

## Abstract (arXiv)

Prior studies on Visual Sentiment Understanding (VSU) primarily rely on the explicit scene information (e.g., facial expression) to judge visual sentiments, which largely ignore implicit scene information (e.g., human action, objection relation and visual background), while such information is critical for precisely discovering visual sentiments. Motivated by this, this paper proposes a new Omni-scene driven visual Sentiment Identifying, Locating and Attributing in videos (Omni-SILA) task, aiming to interactively and precisely identify, locate and attribute visual sentiments through both explicit and implicit scene information. Furthermore, this paper believes that this Omni-SILA task faces two key challenges: modeling scene and highlighting implicit scene beyond explicit. To this end, this paper proposes an Implicit-enhanced Causal MoE (ICM) approach for addressing the Omni-SILA task. Specifically, a Scene-Balanced MoE (SBM) and an Implicit-Enhanced Causal (IEC) blocks are tailored to model scene information and highlight the implicit scene information beyond explicit, respectively. Extensive experimental results on our constructed explicit and implicit Omni-SILA datasets demonstrate the great advantage of the proposed ICM approach over advanced Video-LLMs.