# Efficient Video Summarization Framework using EEG and Eye-tracking Signals

arXiv: https://arxiv.org/abs/2101.11249 · HTML: https://mlsanity.com/paper/2101.11249/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 51 references that are in the map.
- First posted on arXiv 27 Jan 2021.

## Builds on (most cited first, 4 of 4)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [Video Summarization with Attention-Based Encoder-Decoder Networks](https://mlsanity.com/paper/1708.09545/) (2017)
- [Heterogeneous Knowledge Transfer in Video Emotion Recognition, Attribution and Summarization](https://mlsanity.com/paper/1511.04798/) (2015)

## Abstract (arXiv)

This paper proposes an efficient video summarization framework that will give a gist of the entire video in a few key-frames or video skims. Existing video summarization frameworks are based on algorithms that utilize computer vision low-level feature extraction or high-level domain level extraction. However, being the ultimate user of the summarized video, humans remain the most neglected aspect. Therefore, the proposed paper considers human's role in summarization and introduces human visual attention-based summarization techniques. To understand human attention behavior, we have designed and performed experiments with human participants using electroencephalogram (EEG) and eye-tracking technology. The EEG and eye-tracking data obtained from the experimentation are processed simultaneously and used to segment frames containing useful information from a considerable video volume. Thus, the frame segmentation primarily relies on the cognitive judgments of human beings. Using our approach, a video is summarized by 96.5% while maintaining higher precision and high recall factors. The comparison with the state-of-the-art techniques demonstrates that the proposed approach yields ceiling-level performance with reduced computational cost in summarising the videos.