# Consistency Guided Knowledge Retrieval and Denoising in LLMs for Zero-shot Document-level Relation Triplet Extraction

arXiv: https://arxiv.org/abs/2401.13598 · HTML: https://mlsanity.com/paper/2401.13598/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 24 of 24)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [DocRED: A Large-Scale Document-Level Relation Extraction Dataset](https://mlsanity.com/paper/1906.06127/) (2019)
- [Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling](https://mlsanity.com/paper/2010.11304/) (2020)
- [Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme](https://mlsanity.com/paper/1706.05075/) (2017)
- [Double Graph Based Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2009.13752/) (2020)
- [Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.03659/) (2021)
- [Connecting the Dots: Document-level Neural Relation Extraction with Edge-oriented Graphs](https://mlsanity.com/paper/1909.00228/) (2019)
- [RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction](https://mlsanity.com/paper/2203.09101/) (2022)
- [Revisiting DocRED -- Addressing the False Negative Problem in Relation Extraction](https://mlsanity.com/paper/2205.12696/) (2022)
- [Inter-sentence Relation Extraction with Document-level Graph Convolutional Neural Network](https://mlsanity.com/paper/1906.04684/) (2019)
- [A Hierarchical Framework for Relation Extraction with Reinforcement Learning](https://mlsanity.com/paper/1811.03925/) (2018)
- [ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning](https://mlsanity.com/paper/2104.04697/) (2021)
- [Global-to-Local Neural Networks for Document-Level Relation Extraction](https://mlsanity.com/paper/2009.10359/) (2020)
- [An End-to-end Model for Entity-level Relation Extraction using Multi-instance Learning](https://mlsanity.com/paper/2102.05980/) (2021)
- [SIRE: Separate Intra- and Inter-sentential Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2106.01709/) (2021)
- [Discriminative Reasoning for Document-level Relation Extraction](https://mlsanity.com/paper/2106.01562/) (2021)
- [A sequence-to-sequence approach for document-level relation extraction](https://mlsanity.com/paper/2204.01098/) (2022)
- [Three Sentences Are All You Need: Local Path Enhanced Document Relation Extraction](https://mlsanity.com/paper/2106.01793/) (2021)
- [RE-Matching: A Fine-Grained Semantic Matching Method for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2306.04954/) (2023)
- [Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion](https://mlsanity.com/paper/2010.06804/) (2020)
- [None Class Ranking Loss for Document-Level Relation Extraction](https://mlsanity.com/paper/2205.00476/) (2022)

## Built on by (most cited first, 14 of 14)

- [GLiREL -- Generalist Model for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2501.03172/) (2025)
- [RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements](https://mlsanity.com/paper/2411.15700/) (2024)
- [Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extraction](https://mlsanity.com/paper/2402.11142/) (2024)
- [LLM with Relation Classifier for Document-Level Relation Extraction](https://mlsanity.com/paper/2408.13889/) (2024)
- [Scaling Knowledge Graph Construction through Synthetic Data Generation and Distillation](https://mlsanity.com/paper/2410.16597/) (2024)
- [Keyword-driven Retrieval-Augmented Large Language Models for Cold-start User Recommendations](https://mlsanity.com/paper/2405.19612/) (2024)
- [MultiRAG: A Knowledge-guided Framework for Mitigating Hallucination in Multi-source Retrieval Augmented Generation](https://mlsanity.com/paper/2508.03553/) (2025)
- [Automating Intervention Discovery from Scientific Literature: A Progressive Ontology Prompting and Dual-LLM Framework](https://mlsanity.com/paper/2409.00054/) (2024)
- [Biomedical Relation Extraction via Adaptive Document-Relation Cross-Mapping and Concept Unique Identifier](https://mlsanity.com/paper/2501.05155/) (2025)
- [GLiNER-Relex: A Unified Framework for Joint Named Entity Recognition and Relation Extraction](https://mlsanity.com/paper/2605.10108/) (2026)
- [VaeDiff-DocRE: End-to-end Data Augmentation Framework for Document-level Relation Extraction](https://mlsanity.com/paper/2412.13503/) (2024)
- [Relation as a Prior: A Novel Paradigm for LLM-based Document-level Relation Extraction](https://mlsanity.com/paper/2511.08143/) (2025)
- [Document-Level Zero-Shot Relation Extraction with Entity Side Information](https://mlsanity.com/paper/2601.07271/) (2026)
- [CONSISTRE: A Unified Consistency-Aware Framework for Document-Level Relation Extraction with Large Language Models](https://mlsanity.com/paper/2607.24312/) (2026)

## Abstract (arXiv)

Document-level Relation Triplet Extraction (DocRTE) is a fundamental task in information systems that aims to simultaneously extract entities with semantic relations from a document. Existing methods heavily rely on a substantial amount of fully labeled data. However, collecting and annotating data for newly emerging relations is time-consuming and labor-intensive. Recent advanced Large Language Models (LLMs), such as ChatGPT and LLaMA, exhibit impressive long-text generation capabilities, inspiring us to explore an alternative approach for obtaining auto-labeled documents with new relations. In this paper, we propose a Zero-shot Document-level Relation Triplet Extraction (ZeroDocRTE) framework, which generates labeled data by retrieval and denoising knowledge from LLMs, called GenRDK. Specifically, we propose a chain-of-retrieval prompt to guide ChatGPT to generate labeled long-text data step by step. To improve the quality of synthetic data, we propose a denoising strategy based on the consistency of cross-document knowledge. Leveraging our denoised synthetic data, we proceed to fine-tune the LLaMA2-13B-Chat for extracting document-level relation triplets. We perform experiments for both zero-shot document-level relation and triplet extraction on two public datasets. The experimental results illustrate that our GenRDK framework outperforms strong baselines.