# LLM with Relation Classifier for Document-Level Relation Extraction arXiv: https://arxiv.org/abs/2408.13889 ยท HTML: https://mlsanity.com/paper/2408.13889/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 44 of its 59 references that are in the map. - First posted on arXiv 25 Aug 2024. ## Builds on (most cited first, 30 of 44) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021) - [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021) - [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021) - [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021) - [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021) - [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018) - [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019) - [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021) - [Simple BERT Models for Relation Extraction and Semantic Role Labeling](https://mlsanity.com/paper/1904.05255/) (2019) - [Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling](https://mlsanity.com/paper/2010.11304/) (2020) - [Revisiting Relation Extraction in the era of Large Language Models](https://mlsanity.com/paper/2305.05003/) (2023) - [Reasoning with Latent Structure Refinement for Document-Level Relation Extraction](https://mlsanity.com/paper/2005.06312/) (2020) - [Double Graph Based Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2009.13752/) (2020) - [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022) - [Coreferential Reasoning Learning for Language Representation](https://mlsanity.com/paper/2004.06870/) (2020) - [Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs](https://mlsanity.com/paper/1909.00228/) (2019) - [Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extraction](https://mlsanity.com/paper/2102.10249/) (2021) - [Distant Supervision for Relation Extraction beyond the Sentence Boundary](https://mlsanity.com/paper/1609.04873/) (2016) - [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023) - [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022) - [Document-Level N-ary Relation Extraction with Multiscale Representation Learning](https://mlsanity.com/paper/1904.02347/) (2019) - [Simultaneously Self-Attending to All Mentions for Full-Abstract Biological Relation Extraction](https://mlsanity.com/paper/1802.10569/) (2018) ## Built on by (most cited first, 8 of 8) - [EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs](https://mlsanity.com/paper/2505.17139/) (2025) - [Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognition](https://mlsanity.com/paper/2602.04486/) (2026) - [Can an LLM Induce a Graph? Investigating Memory Drift and Context Length](https://mlsanity.com/paper/2510.03611/) (2025) - [GLiNER-Relex: A Unified Framework for Joint Named Entity Recognition and Relation Extraction](https://mlsanity.com/paper/2605.10108/) (2026) - [Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement](https://mlsanity.com/paper/2508.14391/) (2025) - [Relation as a Prior: A Novel Paradigm for LLM-based Document-level Relation Extraction](https://mlsanity.com/paper/2511.08143/) (2025) - [CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models](https://mlsanity.com/paper/2607.24312/) (2026) - [H2: A Dual Hybrid Semantic Data Lake Architecture for Medical Data Harmonization with Human-In-the-Loop verified, LLM Driven Metadata Annotation System](https://mlsanity.com/paper/2608.08056/) (2026) ## Abstract (arXiv) Large language models (LLMs) have created a new paradigm for natural language processing. Despite their advancement, LLM-based methods still lag behind traditional approaches in document-level relation extraction (DocRE), a critical task for understanding complex entity relations within long context. This paper investigates the causes of this performance gap, identifying the dispersion of attention by LLMs due to entity pairs without relations as a key factor. We then introduce a novel classifier-LLM approach to DocRE. Particularly, the proposed approach begins with a classifier designed to select entity pair candidates that exhibit potential relations and then feed them to LLM for final relation classification. This method ensures that the LLM's attention is directed at relation-expressing entity pairs instead of those without relations during inference. Experiments on DocRE benchmarks reveal that our method significantly outperforms recent LLM-based DocRE models and narrows the performance gap with state-of-the-art BERT-based models.