# Benchmarking Large Language Models on Reference Extraction and Parsing in the Social Sciences and Humanities

arXiv: https://arxiv.org/abs/2603.13651 · HTML: https://mlsanity.com/paper/2603.13651/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 17 references that are in the map.
- First posted on arXiv 13 Mar 2026.

## Builds on (most cited first, 5 of 5)

- [Rethinking VLMs and LLMs for Image Classification](https://mlsanity.com/paper/2410.14690/) (2024)
- [A Comparative Study of PDF Parsing Tools Across Diverse Document Categories](https://mlsanity.com/paper/2410.09871/) (2024)
- [Citation Parsing and Analysis with Language Models](https://mlsanity.com/paper/2505.15948/) (2025)
- [The case for the Humanities Citation Index (HuCI): a citation index by the humanities, for the humanities](https://mlsanity.com/paper/2110.00307/) (2021)
- [EXmatcher: Combining Features Based on Reference Strings and Segments to Enhance Citation Matching](https://mlsanity.com/paper/1906.04484/) (2019)

## Abstract (arXiv)

Bibliographic reference extraction and parsing are foundational for citation indexing, linking, and downstream scholarly knowledge-graph construction. However, most established evaluations focus on clean, English, end-of-document bibliographies, and therefore underrepresent the Social Sciences and Humanities (SSH), where citations are frequently multilingual, embedded in footnotes, abbreviated, and shaped by heterogeneous historical conventions. We present a unified benchmark that targets these SSH-realistic conditions across three complementary datasets: CEX (English journal articles spanning multiple disciplines), EXCITE (German/English documents with end-section, footnote-only, and mixed regimes), and LinkedBooks (humanities references with strong stylistic variation and multilinguality). We evaluate three tasks of increasing difficulty -- reference extraction, reference parsing, and end-to-end document parsing -- under a schema-constrained setup that enables direct comparison between a strong supervised pipeline baseline (GROBID) and contemporary LLMs (DeepSeek-V3.1, Mistral-Small-3.2-24B, Gemma-3-27B-it, and Qwen3-VL (4B-32B variants)). Across datasets, extraction largely saturates beyond a moderate capability threshold, while parsing and end-to-end parsing remain the primary bottlenecks due to structured-output brittleness under noisy layouts. We further show that lightweight LoRA adaptation yields consistent gains -- especially on SSH-heavy benchmarks -- and that segmentation/pipelining can substantially improve robustness. Finally, we argue for hybrid deployment via routing: leveraging GROBID for well-structured, in-distribution PDFs while escalating multilingual and footnote-heavy documents to task-adapted LLMs.