# LLVMs4Protest: Harnessing the Power of Large Language and Vision Models for Deciphering Protests in the News

arXiv: https://arxiv.org/abs/2311.18241 · HTML: https://mlsanity.com/paper/2311.18241/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 11 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 6 of 6)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Protest Activity Detection and Perceived Violence Estimation from Social Media Images](https://mlsanity.com/paper/1709.06204/) (2017)
- [Overview of CLEF 2019 Lab ProtestNews: Extracting Protests from News in a Cross-context Setting](https://mlsanity.com/paper/2008.00345/) (2020)
- [Generative AI has lowered the barriers to computational social sciences](https://mlsanity.com/paper/2311.10833/) (2023)

## Abstract (arXiv)

Large language and vision models have transformed how social movements scholars identify protest and extract key protest attributes from multi-modal data such as texts, images, and videos. This article documents how we fine-tuned two large pretrained transformer models, including longformer and swin-transformer v2, to infer potential protests in news articles using textual and imagery data. First, the longformer model was fine-tuned using the Dynamic of Collective Action (DoCA) Corpus. We matched the New York Times articles with the DoCA database to obtain a training dataset for downstream tasks. Second, the swin-transformer v2 models was trained on UCLA-protest imagery data. UCLA-protest project contains labeled imagery data with information such as protest, violence, and sign. Both fine-tuned models will be available via urlhttps://github.com/Joshzyj/llvms4protest. We release this short technical report for social movement scholars who are interested in using LLVMs to infer protests in textual and imagery data.