# Guideline Learning for In-context Information Extraction

arXiv: https://arxiv.org/abs/2310.05066 · HTML: https://mlsanity.com/paper/2310.05066/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 8 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Instruction Induction: From Few Examples to Natural Language Task Descriptions](https://mlsanity.com/paper/2205.10782/) (2022)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [Exploring the Feasibility of ChatGPT for Event Extraction](https://mlsanity.com/paper/2303.03836/) (2023)
- [GPT-RE: In-context Learning for Relation Extraction using Large Language Models](https://mlsanity.com/paper/2305.02105/) (2023)
- [Doc2EDAG: An End-to-End Document-level Framework for Chinese Financial Event Extraction](https://mlsanity.com/paper/1904.07535/) (2019)
- [An Empirical Study on Information Extraction using Large Language Models](https://mlsanity.com/paper/2305.14450/) (2023)
- [Document-level Event Extraction via Heterogeneous Graph-based Interaction Model with a Tracker](https://mlsanity.com/paper/2105.14924/) (2021)
- [Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners](https://mlsanity.com/paper/2210.02969/) (2022)
- [RAAT: Relation-Augmented Attention Transformer for Relation Modeling in Document-Level Event Extraction](https://mlsanity.com/paper/2206.03377/) (2022)
- [Learning In-context Learning for Named Entity Recognition](https://mlsanity.com/paper/2305.11038/) (2023)

## Built on by (most cited first, 24 of 24)

- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling](https://mlsanity.com/paper/2401.14556/) (2024)
- [ADELIE: Aligning Large Language Models on Information Extraction](https://mlsanity.com/paper/2405.05008/) (2024)
- [Instruction-Tuning LLMs for Event Extraction with Annotation Guidelines](https://mlsanity.com/paper/2502.16377/) (2025)
- [ItD: Large Language Models Can Teach Themselves Induction through Deduction](https://mlsanity.com/paper/2403.05789/) (2024)
- [TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generation](https://mlsanity.com/paper/2603.00025/) (2026)
- [Beyond Isolation: Multi-Agent Synergy for Improving Knowledge Graph Construction](https://mlsanity.com/paper/2312.03022/) (2023)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [llmNER: (Zero|Few)-Shot Named Entity Recognition, Exploiting the Power of Large Language Models](https://mlsanity.com/paper/2406.04528/) (2024)
- [GuideX: Guided Synthetic Data Generation for Zero-Shot Information Extraction](https://mlsanity.com/paper/2506.00649/) (2025)
- [Schema as Parameterized Tools for Universal Information Extraction](https://mlsanity.com/paper/2506.01276/) (2025)
- [Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2506.22813/) (2025)
- [Learning Wisdom from Errors: Promoting LLM's Continual Relation Learning through Exploiting Error Cases](https://mlsanity.com/paper/2508.12031/) (2025)
- [Document-Level Tabular Numerical Cross-Checking: A Coarse-to-Fine Approach](https://mlsanity.com/paper/2506.13328/) (2025)
- [Single Ground Truth Is Not Enough: Adding Flexibility to Aspect-Based Sentiment Analysis Evaluation](https://mlsanity.com/paper/2410.09807/) (2024)
- [DiVA-DocRE: A Discriminative and Voice-Aware Paradigm for Document-Level Relation Extraction](https://mlsanity.com/paper/2409.13717/) (2024)
- [Keyword-Centric Prompting for One-Shot Event Detection with Self-Generated Rationale Enhancements](https://mlsanity.com/paper/2508.07598/) (2025)
- [Exploring the Performance of Large Language Models on Subjective Span Identification Tasks](https://mlsanity.com/paper/2601.00736/) (2026)
- [TCM-DiffRAG: Personalized Syndrome Differentiation Reasoning Method for Traditional Chinese Medicine based on Knowledge Graph and Chain of Thought](https://mlsanity.com/paper/2602.22828/) (2026)
- [ProUIE: A Macro-to-Micro Progressive Learning Method for LLM-based Universal Information Extraction](https://mlsanity.com/paper/2604.10633/) (2026)
- [EviSearch: Trustworthy Extraction and Synthesis of Clinical Trial Evidence with Agents that Improve with Use](https://mlsanity.com/paper/2604.14165/) (2026)
- [Beyond Clean Text: Evaluating Encoder and Decoder Robustness for Bangla Event Detection in Noisy Text](https://mlsanity.com/paper/2606.30914/) (2026)
- [One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extraction](https://mlsanity.com/paper/2609.21626/) (2026)

## Abstract (arXiv)

Large language models (LLMs) can perform a new task by merely conditioning on task instructions and a few input-output examples, without optimizing any parameters. This is called In-Context Learning (ICL). In-context Information Extraction (IE) has recently garnered attention in the research community. However, the performance of In-context IE generally lags behind the state-of-the-art supervised expert models. We highlight a key reason for this shortfall: underspecified task description. The limited-length context struggles to thoroughly express the intricate IE task instructions and various edge cases, leading to misalignment in task comprehension with humans. In this paper, we propose a Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines. During the learning phrase, GL automatically synthesizes a set of guidelines based on a few error cases, and during inference, GL retrieves helpful guidelines for better ICL. Moreover, we propose a self-consistency-based active learning method to enhance the efficiency of GL. Experiments on event extraction and relation extraction show that GL can significantly improve the performance of in-context IE.