# Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression

arXiv: https://arxiv.org/abs/2406.19528 · HTML: https://mlsanity.com/paper/2406.19528/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 27 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis](https://mlsanity.com/paper/2309.10771/) (2023)
- [Shaping the Emerging Norms of Using Large Language Models in Social Computing Research](https://mlsanity.com/paper/2307.04280/) (2023)
- [Advancing a Consent-Forward Paradigm for Digital Mental Health Data](https://mlsanity.com/paper/2404.14548/) (2024)
- [#StayHome #WithMe: How Do YouTubers Help with COVID-19 Loneliness?](https://mlsanity.com/paper/2101.03706/) (2021)
- [Understanding and Facilitating Mental Health Help-Seeking of Young Adults: A Socio-technical Ecosystem Framework](https://mlsanity.com/paper/2401.08994/) (2024)

## Built on by (most cited first, 4 of 4)

- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [SafeVid: Toward Safety Aligned Video Large Multimodal Models](https://mlsanity.com/paper/2505.11926/) (2025)
- [Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness](https://mlsanity.com/paper/2504.16936/) (2025)
- [LLM-TA: An LLM-Enhanced Thematic Analysis Pipeline for Transcripts from Parents of Children with Congenital Heart Disease](https://mlsanity.com/paper/2502.01620/) (2025)

## Abstract (arXiv)

Despite the growing interest in leveraging Large Language Models (LLMs) for content analysis, current studies have primarily focused on text-based content. In the present work, we explored the potential of LLMs in assisting video content analysis by conducting a case study that followed a new workflow of LLM-assisted multimodal content analysis. The workflow encompasses codebook design, prompt engineering, LLM processing, and human evaluation. We strategically crafted annotation prompts to get LLM Annotations in structured form and explanation prompts to generate LLM Explanations for a better understanding of LLM reasoning and transparency. To test LLM's video annotation capabilities, we analyzed 203 keyframes extracted from 25 YouTube short videos about depression. We compared the LLM Annotations with those of two human coders and found that LLM has higher accuracy in object and activity Annotations than emotion and genre Annotations. Moreover, we identified the potential and limitations of LLM's capabilities in annotating videos. Based on the findings, we explore opportunities and challenges for future research and improvements to the workflow. We also discuss ethical concerns surrounding future studies based on LLM-assisted video analysis.