# Revisiting Large Language Models as Zero-shot Relation Extractors

arXiv: https://arxiv.org/abs/2310.05028 · HTML: https://mlsanity.com/paper/2310.05028/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 50 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Zero-Shot Relation Extraction via Reading Comprehension](https://mlsanity.com/paper/1706.04115/) (2017)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Enhanced LSTM for Natural Language Inference](https://mlsanity.com/paper/1609.06038/) (2016)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Ask Me Anything: A simple strategy for prompting language models](https://mlsanity.com/paper/2210.02441/) (2022)
- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023)
- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Enriching Pre-trained Language Model with Entity Information for Relation Classification](https://mlsanity.com/paper/1905.08284/) (2019)
- [An Improved Baseline for Sentence-level Relation Extraction](https://mlsanity.com/paper/2102.01373/) (2021)

## Built on by (most cited first, 29 of 29)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward](https://mlsanity.com/paper/2411.03866/) (2024)
- [UrbanKGent: A Unified Large Language Model Agent Framework for Urban Knowledge Graph Construction](https://mlsanity.com/paper/2402.06861/) (2024)
- [Research Trends for the Interplay between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2406.08223/) (2024)
- [Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractors](https://mlsanity.com/paper/2404.17807/) (2024)
- [A Survey on Retrieval And Structuring Augmented Generation with Large Language Models](https://mlsanity.com/paper/2509.10697/) (2025)
- [GenRES: Rethinking Evaluation for Generative Relation Extraction in the Era of Large Language Models](https://mlsanity.com/paper/2402.10744/) (2024)
- [Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning](https://mlsanity.com/paper/2506.05671/) (2025)
- [Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction](https://mlsanity.com/paper/2312.03022/) (2023)
- [Recall, Retrieve and Reason: Towards Better In-Context Relation Extraction](https://mlsanity.com/paper/2404.17809/) (2024)
- [Empirical Analysis of Dialogue Relation Extraction with Large Language Models](https://mlsanity.com/paper/2404.17802/) (2024)
- [Graphical Reasoning: LLM-based Semi-Open Relation Extraction](https://mlsanity.com/paper/2405.00216/) (2024)
- [Domain-Hierarchy Adaptation via Chain of Iterative Reasoning for Few-shot Hierarchical Text Classification](https://mlsanity.com/paper/2407.08959/) (2024)
- [Zero-Shot Classification of Crisis Tweets Using Instruction-Finetuned Large Language Models](https://mlsanity.com/paper/2410.00182/) (2024)
- [A Benchmark for End-to-End Zero-Shot Biomedical Relation Extraction with LLMs: Experiments with OpenAI Models](https://mlsanity.com/paper/2504.04083/) (2025)
- [Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R2)GRPO](https://mlsanity.com/paper/2505.22068/) (2025)
- [Enhancing Multilingual LLM-based ASR with Mixture of Experts and Dynamic Downsampling](https://mlsanity.com/paper/2606.10439/) (2026)
- [Summarization for Generative Relation Extraction in the Microbiome Domain](https://mlsanity.com/paper/2506.08647/) (2025)
- [Customizing Speech Recognition Model with Large Language Model Feedback](https://mlsanity.com/paper/2506.11091/) (2025)
- [What's on My Network? Using Large Language Models to Identify Real-World IoT Devices at Scale](https://mlsanity.com/paper/2510.13817/) (2025)
- [Relation Extraction with Instance-Adapted Predicate Descriptions](https://mlsanity.com/paper/2503.17799/) (2025)
- [Spatial Language Likelihood Grounding Network for Bayesian Fusion of Human-Robot Observations](https://mlsanity.com/paper/2507.19947/) (2025)
- [Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement](https://mlsanity.com/paper/2508.14391/) (2025)
- [MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning](https://mlsanity.com/paper/2601.07005/) (2026)
- [A Study on Building Efficient Zero-Shot Relation Extraction Models](https://mlsanity.com/paper/2603.01266/) (2026)
- [Relational Probing: LM-to-Graph Adaptation for Financial Prediction](https://mlsanity.com/paper/2604.10212/) (2026)
- [Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction](https://mlsanity.com/paper/2606.22606/) (2026)
- [Mapping Political-Elite Networks in Europe with a Multilingual Joint Entity-Relation Extraction Pipeline](https://mlsanity.com/paper/2606.27347/) (2026)
- [ANCHOR-RE: An Agentic Neuro-Symbolic Framework for Grounded Biomedical Relation Extraction](https://mlsanity.com/paper/2608.03154/) (2026)

## Abstract (arXiv)

Relation extraction (RE) consistently involves a certain degree of labeled or unlabeled data even if under zero-shot setting. Recent studies have shown that large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt, which provides the possibility of extracting relations from text without any data and parameter tuning. This work focuses on the study of exploring LLMs, such as ChatGPT, as zero-shot relation extractors. On the one hand, we analyze the drawbacks of existing RE prompts and attempt to incorporate recent prompt techniques such as chain-of-thought (CoT) to improve zero-shot RE. We propose the summarize-and-ask (textscSumAsk) prompting, a simple prompt recursively using LLMs to transform RE inputs to the effective question answering (QA) format. On the other hand, we conduct comprehensive experiments on various benchmarks and settings to investigate the capabilities of LLMs on zero-shot RE. Specifically, we have the following findings: (i) textscSumAsk consistently and significantly improves LLMs performance on different model sizes, benchmarks and settings; (ii) Zero-shot prompting with ChatGPT achieves competitive or superior results compared with zero-shot and fully supervised methods; (iii) LLMs deliver promising performance in extracting overlapping relations; (iv) The performance varies greatly regarding different relations. Different from small language models, LLMs are effective in handling challenge none-of-the-above (NoTA) relation.