# Improving Recall of Large Language Models: A Model Collaboration Approach for Relational Triple Extraction

arXiv: https://arxiv.org/abs/2404.09593 · HTML: https://mlsanity.com/paper/2404.09593/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 15 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Does Synthetic Data Generation of LLMs Help Clinical Text Mining?](https://mlsanity.com/paper/2303.04360/) (2023)
- [ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports](https://mlsanity.com/paper/2212.14882/) (2022)
- [Entity-Relation Extraction as Multi-Turn Question Answering](https://mlsanity.com/paper/1905.05529/) (2019)
- [Exploring the Feasibility of ChatGPT for Event Extraction](https://mlsanity.com/paper/2303.03836/) (2023)
- [TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking](https://mlsanity.com/paper/2010.13415/) (2020)
- [Zero-shot Temporal Relation Extraction with ChatGPT](https://mlsanity.com/paper/2304.05454/) (2023)
- [A Hierarchical Framework for Relation Extraction with Reinforcement Learning](https://mlsanity.com/paper/1811.03925/) (2018)
- [Joint Extraction of Entities and Relations Based on a Novel Decomposition Strategy](https://mlsanity.com/paper/1909.04273/) (2019)
- [A Simple Zero-shot Prompt Weighting Technique to Improve Prompt Ensembling in Text-Image Models](https://mlsanity.com/paper/2302.06235/) (2023)
- [CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases](https://mlsanity.com/paper/1610.08763/) (2016)
- [Revisiting the Negative Data of Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2105.10158/) (2021)
- [Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach](https://mlsanity.com/paper/1707.00166/) (2017)
- [ZS4IE: A toolkit for Zero-Shot Information Extraction with simple Verbalizations](https://mlsanity.com/paper/2203.13602/) (2022)
- [InSRL: A Multi-view Learning Framework Fusing Multiple Information Sources for Distantly-supervised Relation Extraction](https://mlsanity.com/paper/2012.09370/) (2020)

## Built on by (most cited first, 3 of 3)

- [Traceable LLM-based validation of statements in knowledge graphs](https://mlsanity.com/paper/2409.07507/) (2024)
- [Adaptive Reinforcement Learning Planning: Harnessing Large Language Models for Complex Information Extraction](https://mlsanity.com/paper/2406.11455/) (2024)
- [RLAP: A Reinforcement Learning Enhanced Adaptive Planning Framework for Multi-step NLP Task Solving](https://mlsanity.com/paper/2505.11893/) (2025)

## Abstract (arXiv)

Relation triple extraction, which outputs a set of triples from long sentences, plays a vital role in knowledge acquisition. Large language models can accurately extract triples from simple sentences through few-shot learning or fine-tuning when given appropriate instructions. However, they often miss out when extracting from complex sentences. In this paper, we design an evaluation-filtering framework that integrates large language models with small models for relational triple extraction tasks. The framework includes an evaluation model that can extract related entity pairs with high precision. We propose a simple labeling principle and a deep neural network to build the model, embedding the outputs as prompts into the extraction process of the large model. We conduct extensive experiments to demonstrate that the proposed method can assist large language models in obtaining more accurate extraction results, especially from complex sentences containing multiple relational triples. Our evaluation model can also be embedded into traditional extraction models to enhance their extraction precision from complex sentences.