# From Variance to Invariance: Qualitative Content Analysis for Narrative Graph Annotation

arXiv: https://arxiv.org/abs/2603.01930 · HTML: https://mlsanity.com/paper/2603.01930/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 39 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 4 of 4)

- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Where Do People Tell Stories Online? Story Detection Across Online Communities](https://mlsanity.com/paper/2311.09675/) (2023)
- [Can LLMs Learn Macroeconomic Narratives from Social Media?](https://mlsanity.com/paper/2406.12109/) (2024)
- [Text Annotation via Inductive Coding: Comparing Human Experts to LLMs in Qualitative Data Analysis](https://mlsanity.com/paper/2512.00046/) (2025)

## Abstract (arXiv)

Narratives in news discourse play a critical role in shaping public understanding of economic events, such as inflation. Annotating and evaluating these narratives in a structured manner remains a key challenge for Natural Language Processing (NLP). In this work, we introduce a narrative graph annotation framework that integrates principles from qualitative content analysis (QCA) to prioritize annotation quality by reducing annotation errors. We present a dataset of inflation narratives annotated as directed acyclic graphs (DAGs), where nodes represent events and edges encode causal relations. To evaluate annotation quality, we employed a 6×3 factorial experimental design to examine the effects of narrative representation (six levels) and distance metric type (three levels) on inter-annotator agreement (Krippendorrf's α), capturing the presence of human label variation (HLV) in narrative interpretations. Our analysis shows that (1) lenient metrics (overlap-based distance) overestimate reliability, and (2) locally-constrained representations (e.g., one-hop neighbors) reduce annotation variability. Our annotation and implementation of graph-based Krippendorrf's α are open-sourced. The annotation framework and evaluation results provide practical guidance for NLP research on graph-based narrative annotation under HLV.