# A Hierarchical Error Framework for Reliable Automated Coding in Communication Research: Applications to Health and Political Communication

arXiv: https://arxiv.org/abs/2509.24841 · HTML: https://mlsanity.com/paper/2509.24841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 12 of 12)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Automated Clinical Coding: What, Why, and Where We Are?](https://mlsanity.com/paper/2203.11092/) (2022)
- [Learning Hierarchy Aware Features for Reducing Mistake Severity](https://mlsanity.com/paper/2207.12646/) (2022)
- [The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review](https://mlsanity.com/paper/2402.13635/) (2024)
- [Stability Analysis of ChatGPT-based Sentiment Analysis in AI Quality Assurance](https://mlsanity.com/paper/2401.07441/) (2024)
- [A Confidence-Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks](https://mlsanity.com/paper/2508.02029/) (2025)
- [Automated Quality Assessment for LLM-Based Complex Qualitative Coding: A Confidence-Diversity Framework](https://mlsanity.com/paper/2508.20462/) (2025)

## Abstract (arXiv)

Automated content analysis increasingly supports communication research, yet scaling manual coding into computational pipelines raises concerns about measurement reliability and validity. We introduce a Hierarchical Error Correction (HEC) framework that treats model failures as layered measurement errors (knowledge gaps, reasoning limitations, and complexity constraints) and targets the layers that most affect inference. The framework implements a three-phase methodology: systematic error profiling across hierarchical layers, targeted intervention design matched to dominant error sources, and rigorous validation with statistical testing. Evaluating HEC across health communication (medical specialty classification) and political communication (bias detection), and legal tasks, we validate the approach with five diverse large language models. Results show average accuracy gains of 11.2 percentage points (p < .001, McNemar's test) and stable conclusions via reduced systematic misclassification. Cross-model validation demonstrates consistent improvements (range: +6.8 to +14.6pp), with effectiveness concentrated in moderate-to-high baseline tasks (50-85% accuracy). A boundary study reveals diminished returns in very high-baseline (>85%) or precision-matching tasks, establishing applicability limits. We map layered errors to threats to construct and criterion validity and provide a transparent, measurement-first blueprint for diagnosing error profiles, selecting targeted interventions, and reporting reliability/validity evidence alongside accuracy. This applies to automated coding across communication research and the broader social sciences.