# Corpus-Level Evaluation for Event QA: The IndiaPoliceEvents Corpus Covering the 2002 Gujarat Violence

arXiv: https://arxiv.org/abs/2105.12936 · HTML: https://mlsanity.com/paper/2105.12936/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 96 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Event Extraction by Answering (Almost) Natural Questions](https://mlsanity.com/paper/2004.13625/) (2020)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://mlsanity.com/paper/2010.08240/) (2020)
- [Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning](https://mlsanity.com/paper/1912.10389/) (2019)
- [Deeper Text Understanding for IR with Contextual Neural Language Modeling](https://mlsanity.com/paper/1905.09217/) (2019)
- [CEDR: Contextualized Embeddings for Document Ranking](https://mlsanity.com/paper/1904.07094/) (2019)
- [Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpus](https://mlsanity.com/paper/2105.05241/) (2021)
- [RepBERT: Contextualized Text Embeddings for First-Stage Retrieval](https://mlsanity.com/paper/2006.15498/) (2020)
- [Large-Scale QA-SRL Parsing](https://mlsanity.com/paper/1805.05377/) (2018)
- [Controlled Crowdsourcing for High-Quality QA-SRL Annotation](https://mlsanity.com/paper/1911.03243/) (2019)
- [A Survey on Explainability in Machine Reading Comprehension](https://mlsanity.com/paper/2010.00389/) (2020)
- [Evidence Sentence Extraction for Machine Reading Comprehension](https://mlsanity.com/paper/1902.08852/) (2019)
- [Crowdsourcing Ground Truth for Medical Relation Extraction](https://mlsanity.com/paper/1701.02185/) (2017)
- [Identifying civilians killed by police with distantly supervised entity-event extraction](https://mlsanity.com/paper/1707.07086/) (2017)
- [Who is Killed by Police: Introducing Supervised Attention for Hierarchical LSTMs](https://mlsanity.com/paper/1807.03409/) (2018)
- [Uncertainty over Uncertainty: Investigating the Assumptions, Annotations, and Text Measurements of Economic Policy Uncertainty](https://mlsanity.com/paper/2010.04706/) (2020)
- [The Simplest Thing That Can Possibly Work: Pseudo-Relevance Feedback Using Text Classification](https://mlsanity.com/paper/1904.08861/) (2019)
- [Generating Politically-Relevant Event Data](https://mlsanity.com/paper/1609.06239/) (2016)

## Built on by (most cited first, 19 of 19)

- [Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts](https://mlsanity.com/paper/2407.10747/) (2024)
- [Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive](https://mlsanity.com/paper/2301.12534/) (2023)
- [Synthetically generated text for supervised text analysis](https://mlsanity.com/paper/2303.16028/) (2023)
- [Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2021): Workshop and Shared Task Report](https://mlsanity.com/paper/2108.07865/) (2021)
- [Political DEBATE: Efficient Zero-shot and Few-shot Classifiers for Political Text](https://mlsanity.com/paper/2409.02078/) (2024)
- [Rethinking the Event Coding Pipeline with Prompt Entailment](https://mlsanity.com/paper/2210.05257/) (2022)
- [Event Coreference Resolution for Contentious Politics Events](https://mlsanity.com/paper/2203.10123/) (2022)
- [Infrastructure Ombudsman: Mining Future Failure Concerns from Structural Disaster Response](https://mlsanity.com/paper/2402.13528/) (2024)
- [Disentangling Societal Inequality from Model Biases: Gender Inequality in Divorce Court Proceedings](https://mlsanity.com/paper/2307.10200/) (2023)
- [Creating Custom Event Data Without Dictionaries: A Bag-of-Tricks](https://mlsanity.com/paper/2304.01331/) (2023)
- [Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speech](https://mlsanity.com/paper/2502.09004/) (2025)
- [What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification](https://mlsanity.com/paper/2510.03541/) (2025)
- [Zero-Shot Ranking Socio-Political Texts with Transformer Language Models to Reduce Close Reading Time](https://mlsanity.com/paper/2210.09179/) (2022)
- [Op-Fed: Opinion, Stance, and Monetary Policy Annotations on FOMC Transcripts Using Active Learning](https://mlsanity.com/paper/2509.13539/) (2025)
- [Transformer Encoder for Social Science](https://mlsanity.com/paper/2208.08005/) (2022)
- [Extracting Victim Counts from Text](https://mlsanity.com/paper/2302.12367/) (2023)
- [Community Needs and Assets: A Computational Analysis of Community Conversations](https://mlsanity.com/paper/2403.13272/) (2024)
- [Class-Aware Contrastive Optimization for Imbalanced Text Classification](https://mlsanity.com/paper/2410.22197/) (2024)
- [Two Centuries of Sexism in British Parliament: A Computational Analysis of Women's Representation in the Hansard Corpus](https://mlsanity.com/paper/2608.30485/) (2026)

## Abstract (arXiv)

Automated event extraction in social science applications often requires corpus-level evaluations: for example, aggregating text predictions across metadata and unbiased estimates of recall. We combine corpus-level evaluation requirements with a real-world, social science setting and introduce the IndiaPoliceEvents corpus--all 21,391 sentences from 1,257 English-language Times of India articles about events in the state of Gujarat during March 2002. Our trained annotators read and label every document for mentions of police activity events, allowing for unbiased recall evaluations. In contrast to other datasets with structured event representations, we gather annotations by posing natural questions, and evaluate off-the-shelf models for three different tasks: sentence classification, document ranking, and temporal aggregation of target events. We present baseline results from zero-shot BERT-based models fine-tuned on natural language inference and passage retrieval tasks. Our novel corpus-level evaluations and annotation approach can guide creation of similar social-science-oriented resources in the future.