Do You See What I See? Capabilities and Limits of Automated Multimedia Content Analysis
Not on the map. Why?
Abstract (arXiv)
The ever-increasing amount of user-generated content online has led, in recent years, to an expansion in research and investment in automated content analysis tools. Scrutiny of automated content analysis has accelerated during the COVID-19 pandemic, as social networking services have placed a greater reliance on these tools due to concerns about health risks to their moderation staff from in-person work. At the same time, there are important policy debates around the world about how to improve content moderation while protecting free expression and privacy. In order to advance these debates, we need to understand the potential role of automated content analysis tools. This paper explains the capabilities and limitations of tools for analyzing online multimedia content and highlights the potential risks of using these tools at scale without accounting for their limitations. It focuses on two main categories of tools: matching models and computer prediction models. Matching models include cryptographic and perceptual hashing, which compare user-generated content with existing and known content. Predictive models (including computer vision and computer audition) are machine learning techniques that aim to identify characteristics of new or previously unknown content.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 49 of its 103 references that are in the map. First posted on arXiv 15 Dec 2021.
Builds on
49 of 103 references.
References · 49
- 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
- 2015Fast R-CNNcited by 2,930
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2018YOLOv3: An Incremental Improvementcited by 2,652
- 2016Pyramid Scene Parsing Networkcited by 2,472
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
- 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
- 2017A systematic study of the class imbalance problem in convolutional neural networkscited by 510
- 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
- 2020Measuring Robustness to Natural Distribution Shifts in Image Classificationcited by 452
37 more
- 20183D human pose estimation in video with temporal convolutions and semi-supervised trainingcited by 401
- 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
- 2019Fooling automated surveillance cameras: adversarial patches to attack person detectioncited by 268
- 2020Large image datasets: A pyrrhic win for computer vision?cited by 237
- 2020Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervisioncited by 235
- 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
- 2016Speed/accuracy trade-offs for modern convolutional object detectorscited by 198
- 2019Learning Individual Styles of Conversational Gesturecited by 190
- 2019One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniquescited by 164
- 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
- 2019Learning from Synthetic Data for Crowd Counting in the Wildcited by 148
- 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
- 2019Classifying, Segmenting, and Tracking Object Instances in Video with Mask Propagationcited by 125
- 2020What Matters in Unsupervised Optical Flowcited by 124
- 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
- 2017Robust Physical-World Attacks on Deep Learning Modelscited by 70
- 2020Detecting Deep-Fake Videos from Appearance and Behaviorcited by 66
- 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
- 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
- 2019Toward a better trade-off between performance and fairness with kernel-based distribution matchingcited by 36
- 2020Deepfakes Detection with Automatic Face Weightingcited by 35
- 2019AdaScale: Towards Real-time Video Object Detection Using Adaptive Scalingcited by 27
- 2018Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Surveycited by 25
- 2020Detecting natural disasters, damage, and incidents in the wildcited by 23
- 2017Protest Activity Detection and Perceived Violence Estimation from Social Media Imagescited by 20
- 2020One-shot conditional audio filtering of arbitrary soundscited by 18
- 2017Understanding and Comparing Deep Neural Networks for Age and Gender Classificationcited by 17
- 2020Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translationcited by 14
- 2016Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detectioncited by 10
- 2017Standard detectors aren't (currently) fooled by physical adversarial stop signscited by 7
- 2020Adversarial collision attacks on image hashing functionscited by 5
- 2019PDQ & TMK + PDQF -- A Test Drive of Facebook's Perceptual Hashing Algorithmscited by 4
- 2018Deep Unsupervised Multi-View Detection of Video Game Stream Highlightscited by 3
- 2020StarNet: towards Weakly Supervised Few-Shot Object Detectioncited by 2
- 2020Metadata-Based Detection of Child Sexual Abuse Materialcited by 2
- 2018Adult content in Social Live Streaming Services: Characterizing deviant users and relationshipscited by 1
- 2020Strategies for Robust Image Classificationcited by 1
Built on by
Citing papers per year: 2023: 2.
2 papers.