# Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty

arXiv: https://arxiv.org/abs/2309.03433 · HTML: https://mlsanity.com/paper/2309.03433/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 12 of 12)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [A Survey on Open Information Extraction](https://mlsanity.com/paper/1806.05599/) (2018)
- [An Empirical Study on Information Extraction using Large Language Models](https://mlsanity.com/paper/2305.14450/) (2023)
- [IMoJIE: Iterative Memory-Based Joint Open Information Extraction](https://mlsanity.com/paper/2005.08178/) (2020)
- [Span Model for Open Information Extraction on Accurate Corpus](https://mlsanity.com/paper/1901.10879/) (2019)
- [A Survey on Neural Open Information Extraction: Current Status and Future Directions](https://mlsanity.com/paper/2205.11725/) (2022)
- [A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises](https://mlsanity.com/paper/2306.04802/) (2023)
- [PIVOINE: Instruction Tuning for Open-world Information Extraction](https://mlsanity.com/paper/2305.14898/) (2023)
- [Knowledge-enhanced Neural Machine Reasoning: A Review](https://mlsanity.com/paper/2302.02093/) (2023)

## Built on by (most cited first, 8 of 8)

- [Uncertainty Quantification for In-Context Learning of Large Language Models](https://mlsanity.com/paper/2402.10189/) (2024)
- [CHisIEC: An Information Extraction Corpus for Ancient Chinese History](https://mlsanity.com/paper/2403.15088/) (2024)
- [Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications](https://mlsanity.com/paper/2507.05517/) (2025)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models](https://mlsanity.com/paper/2406.02148/) (2024)
- [Harnessing the Power of Large Language Model for Uncertainty Aware Graph Processing](https://mlsanity.com/paper/2404.00589/) (2024)
- [Granting GPT-4 License and Opportunity: Enhancing Accuracy and Confidence Estimation for Few-Shot Event Detection](https://mlsanity.com/paper/2408.00914/) (2024)
- [TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation](https://mlsanity.com/paper/2607.03447/) (2026)

## Abstract (arXiv)

Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text, typically in the form of (subject, relation, object) triples. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant context from relevant relations and generate structured output due to the restrictions on fine-tuning the model. Second, LLMs generates responses autoregressively based on probability, which makes the predicted relations lack confidence. In this paper, we assess the capabilities of LLMs in improving the OIE task. Particularly, we propose various in-context learning strategies to enhance LLM's instruction-following ability and a demonstration uncertainty quantification module to enhance the confidence of the generated relations. Our experiments on three OIE benchmark datasets show that our approach holds its own against established supervised methods, both quantitatively and qualitatively.