# EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues

arXiv: https://arxiv.org/abs/2502.02172 · HTML: https://mlsanity.com/paper/2502.02172/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 63 references that are in the map.
- First posted on arXiv 4 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [BoT-SORT: Robust Associations Multi-Pedestrian Tracking](https://mlsanity.com/paper/2206.14651/) (2022)
- [Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection](https://mlsanity.com/paper/2107.06592/) (2021)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)
- [TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection](https://mlsanity.com/paper/1908.05786/) (2019)
- [ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction](https://mlsanity.com/paper/2012.06170/) (2020)
- [STAViS: Spatio-Temporal AudioVisual Saliency Network](https://mlsanity.com/paper/2001.03063/) (2020)
- [Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups](https://mlsanity.com/paper/1605.06489/) (2016)
- [Pano2Vid: Automatic Cinematography for Watching 360circ Videos](https://mlsanity.com/paper/1612.02335/) (2016)
- [Temporal-Spatial Feature Pyramid for Video Saliency Detection](https://mlsanity.com/paper/2105.04213/) (2021)
- [The Emotion Dynamics of Literary Novels](https://mlsanity.com/paper/2403.02474/) (2024)
- [GAZED- Gaze-guided Cinematic Editing of Wide-Angle Monocular Video Recordings](https://mlsanity.com/paper/2010.11886/) (2020)
- [Towards 3D Face Reconstruction in Perspective Projection: Estimating 6DoF Face Pose from Monocular Image](https://mlsanity.com/paper/2205.04126/) (2022)
- [Major Entity Identification: A Generalizable Alternative to Coreference Resolution](https://mlsanity.com/paper/2406.14654/) (2024)
- [Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling](https://mlsanity.com/paper/2401.12039/) (2024)
- [Smart Director: An Event-Driven Directing System for Live Broadcasting](https://mlsanity.com/paper/2201.04024/) (2022)
- [Joint Stabilization and Direction of 360deg Videos](https://mlsanity.com/paper/1901.04161/) (2019)
- [Watch to Edit: Video Retargeting using Gaze](https://mlsanity.com/paper/1807.03125/) (2018)
- [Real Time GAZED: Online Shot Selection and Editing of Virtual Cameras from Wide-Angle Monocular Video Recordings](https://mlsanity.com/paper/2311.15581/) (2023)

## Abstract (arXiv)

We present EditIQ, a completely automated framework for cinematically editing scenes captured via a stationary, large field-of-view and high-resolution camera. From the static camera feed, EditIQ initially generates multiple virtual feeds, emulating a team of cameramen. These virtual camera shots termed rushes are subsequently assembled using an automated editing algorithm, whose objective is to present the viewer with the most vivid scene content. To understand key scene elements and guide the editing process, we employ a two-pronged approach: (1) a large language model (LLM)-based dialogue understanding module to analyze conversational flow, coupled with (2) visual saliency prediction to identify meaningful scene elements and camera shots therefrom. We then formulate cinematic video editing as an energy minimization problem over shot selection, where cinematic constraints determine shot choices, transitions, and continuity. EditIQ synthesizes an aesthetically and visually compelling representation of the original narrative while maintaining cinematic coherence and a smooth viewing experience. Efficacy of EditIQ against competing baselines is demonstrated via a psychophysical study involving twenty participants on the BBC Old School dataset plus eleven theatre performance videos. Video samples from EditIQ can be found at https://editiq-ave.github.io/.