# A Unified View of Evaluation Metrics for Structured Prediction

arXiv: https://arxiv.org/abs/2310.13793 · HTML: https://mlsanity.com/paper/2310.13793/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 47 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Document-Level Event Argument Extraction by Conditional Generation](https://mlsanity.com/paper/2104.05919/) (2021)
- [SciREX: A Challenge Dataset for Document-Level Information Extraction](https://mlsanity.com/paper/2005.00512/) (2020)
- [Cross-Sentence N-ary Relation Extraction with Graph LSTMs](https://mlsanity.com/paper/1708.03743/) (2017)
- [Parallelizing the dual revised simplex method](https://mlsanity.com/paper/1503.01889/) (2015)
- [On Valid Optimal Assignment Kernels and Applications to Graph Classification](https://mlsanity.com/paper/1606.01141/) (2016)
- [Document-level Entity-based Extraction as Template Generation](https://mlsanity.com/paper/2109.04901/) (2021)
- [Document-Level Event Role Filler Extraction using Multi-Granularity Contextualized Encoding](https://mlsanity.com/paper/2005.06579/) (2020)
- [Hierarchical Entity Typing via Multi-level Learning to Rank](https://mlsanity.com/paper/2004.02286/) (2020)
- [Iterative Document-level Information Extraction via Imitation Learning](https://mlsanity.com/paper/2210.06600/) (2022)
- [Automatic Error Analysis for Document-level Information Extraction](https://mlsanity.com/paper/2209.07442/) (2022)

## Built on by (most cited first, 11 of 11)

- [Learning to Retrieve Iteratively for In-Context Learning](https://mlsanity.com/paper/2406.14739/) (2024)
- [WikiVideo: Article Generation from Multiple Videos](https://mlsanity.com/paper/2504.00939/) (2025)
- [Learning Skill-Attributes for Transferable Assessment in Video](https://mlsanity.com/paper/2511.13993/) (2025)
- [MultiMUC: Multilingual Template Filling on MUC-4](https://mlsanity.com/paper/2401.16209/) (2024)
- [The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models](https://mlsanity.com/paper/2604.25359/) (2026)
- [FAMuS: Frames Across Multiple Sources](https://mlsanity.com/paper/2311.05601/) (2023)
- [Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer](https://mlsanity.com/paper/2606.11018/) (2026)
- [Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2510.24870/) (2025)
- [SCP-NL2TL: Selective Conformal Prediction with Semantic Verification for Natural Language to Temporal Logic Specifications](https://mlsanity.com/paper/2608.05439/) (2026)
- [Small Models Are (Still) Effective Cross-Domain Argument Extractors](https://mlsanity.com/paper/2404.08579/) (2024)
- [Cross-Document Event-Keyed Summarization](https://mlsanity.com/paper/2410.14795/) (2024)

## Abstract (arXiv)

We present a conceptual framework that unifies a variety of evaluation metrics for different structured prediction tasks (e.g. event and relation extraction, syntactic and semantic parsing). Our framework requires representing the outputs of these tasks as objects of certain data types, and derives metrics through matching of common substructures, possibly followed by normalization. We demonstrate how commonly used metrics for a number of tasks can be succinctly expressed by this framework, and show that new metrics can be naturally derived in a bottom-up way based on an output structure. We release a library that enables this derivation to create new metrics. Finally, we consider how specific characteristics of tasks motivate metric design decisions, and suggest possible modifications to existing metrics in line with those motivations.