mlsanityData to 14 Oct 2026

GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

arXiv 2310.03668 ↗First posted 5 Oct 2023cs.CLcited by 119

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) combined with instruction tuning have made significant progress when generalizing to unseen tasks. However, they have been less successful in Information Extraction (IE), lagging behind task-specific models. Typically, IE tasks are characterized by complex annotation guidelines that describe the task and give examples to humans. Previous attempts to leverage such information have failed, even with the largest models, as they are not able to follow the guidelines out of the box. In this paper, we propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. Comprehensive evaluation empirically demonstrates that GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous attempts at zero-shot information extraction. The ablation study shows that detailed guidelines are key for good results.

Key facts. Cited by 119 papers in the map (data through 14 Oct 2026). Builds on 45 of its 73 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

45 of 73 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,139
33 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  3. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  8. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  10. 2017Zero-Shot Relation Extraction via Reading Comprehensioncited by 426
  11. 2022Data Contamination: From Memorization to Exploitationcited by 236
  12. 2022Unified Structure Generation for Universal Information Extractioncited by 221
  13. 2021How Many Data Points is a Prompt Worth?cited by 198
  14. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 170
  15. 2021Document-Level Event Argument Extraction by Conditional Generationcited by 155
  16. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 104
  17. 2019Multi-Sentence Argument Linkingcited by 81
  18. 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
  19. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  20. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 72
  21. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  22. 2023CodeIE: Large Code Generation Models are Better Few-Shot Information Extractorscited by 68
  23. 2023Universal Information Extraction as Unified Semantic Matchingcited by 48
  24. 2022Prompting Language Models for Linguistic Structurecited by 39
  25. 2023Language Models are Few-shot Learners for Prognostic Predictioncited by 37
  26. 2023PromptNER: Prompting For Named Entity Recognitioncited by 35
  27. 2020Biomedical Named Entity Recognition at Scalecited by 30
  28. 2022Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learningcited by 25
  29. 2022Optimizing Bi-Encoder for Named Entity Recognition via Contrastive Learningcited by 24
  30. 2023SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)cited by 17
  31. 2023DAMO-NLP at SemEval-2023 Task 2: A Unified Retrieval-augmented System for Multilingual Named Entity Recognitioncited by 9
  32. 2022ZS4IE: A toolkit for Zero-Shot Information Extraction with simple Verbalizationscited by 7
  33. 2023PromptNER: A Prompting Method for Few-shot Named Entity Recognition via k Nearest Neighbor Searchcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 40; 2025: 39; 2026: 36.

119 papers.

Sort:

By area:

  1. 2023Large Language Models for Generative Information Extraction: A Surveycited by 185
  2. 2024Large Language Models Meet NLP: A Surveycited by 103
  3. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 88
  4. 2024NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Datacited by 34
  5. 2024KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extractioncited by 34
  6. 2024Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extractioncited by 33
  7. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  8. 2024C-ICL: Contrastive In-context Learning for Information Extractioncited by 23
  9. 2024SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documentscited by 21
  10. 2025Instruction-Tuning LLMs for Event Extraction with Annotation Guidelinescited by 18
  11. 2024A Simple but Effective Approach to Improve Structured Language Model Output for Information Extractioncited by 16
  12. 2023Self-Improving for Zero-Shot Named Entity Recognition with Large Language Modelscited by 15
38 more
  1. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  2. 2024Rethinking Negative Instances for Generative Named Entity Recognitioncited by 13
  3. 2025GLiREL -- Generalist Model for Zero-Shot Relation Extractioncited by 12
  4. 2024LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Markingcited by 7
  5. 2026TAB-PO: Preference Optimization with a Token-Level Adaptive Barrier for Token-Critical Structured Generationcited by 7
  6. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
  7. 2024Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Modelscited by 6
  8. 2025A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognitioncited by 6
  9. 2025Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extractioncited by 6
  10. 2025CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideationcited by 6
  11. 2025MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extractioncited by 6
  12. 2024IEPile: Unearthing Large-Scale Schema-Based Information Extraction Corpuscited by 5
  13. 2024P-ICL: Point In-Context Learning for Named Entity Recognition with Large Language Modelscited by 5
  14. 2024Beyond Boundaries: Learning a Universal Entity Taxonomy across Datasets and Languages for Open Named Entity Recognitioncited by 5
  15. 2024Show Less, Instruct More: Enriching Prompts with Definitions and Guidelines for Zero-Shot NERcited by 5
  16. 2024Improving Few-Shot Cross-Domain Named Entity Recognition by Instruction Tuning a Word-Embedding based Retrieval Augmented Large Language Modelcited by 5
  17. 2025CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrievalcited by 5
  18. 2025Event Extraction in Large Language Modelcited by 5
  19. 2024An In-depth Evaluation of Large Language Models in Sentence Simplification with Error-based Human Assessmentcited by 4
  20. 2024Curating Grounded Synthetic Data with Global Perspectives for Equitable AIcited by 4
  21. 2024Retrieval Augmented Instruction Tuning for Open NER with Large Language Modelscited by 4
  22. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4
  23. 2025GuideX: Guided Synthetic Data Generation for Zero-Shot Information Extractioncited by 4
  24. 2025Multilingual Dialogue Generation and Localization with Dialogue Act Scriptingcited by 4
  25. 2024EventRL: Enhancing Event Extraction with Outcome Supervision for Large Language Modelscited by 3
  26. 2024Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Modelscited by 3
  27. 2024Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodologycited by 3
  28. 2024WhisperNER: Unified Open Named Entity and Speech Recognitioncited by 3
  29. 2024RUIE: Retrieval-based Unified Information Extraction using Large Language Modelcited by 3
  30. 2024KnowCoder-X: Boosting Multilingual Information Extraction via Codecited by 3
  31. 2024Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Datacited by 3
  32. 2025Schema as Parameterized Tools for Universal Information Extractioncited by 3
  33. 2025Selecting and Merging: Towards Adaptable and Scalable Named Entity Recognition with Large Language Modelscited by 3
  34. 2025GLiDRE: Generalist Lightweight model for Document-level Relation Extractioncited by 3
  35. 2025FiNERweb: Datasets and Artifacts for Scalable Multilingual Named Entity Recognitioncited by 3
  36. 2024Event Extraction in Basque: Typologically motivated Cross-Lingual Transfer-Learning Analysiscited by 2
  37. 2024E2MoCase: A Dataset for Emotional, Event and Moral Observations in News Articles on High-impact Legal Casescited by 2
  38. 2024Beyond the Numbers: Transparency in Relation Extraction Benchmark Creation and Leaderboardscited by 2

50 most cited of 119 All 119 (JSON)