mlsanityData to 7 Oct 2026

Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!

arXiv 2303.08559 ↗First posted 15 Mar 2023cs.CL cs.AIcited by 119

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs are competitive few-shot solvers for information extraction (IE) tasks, however, remains an open problem. In this work, we aim to provide a thorough answer to this question. Through extensive experiments on nine datasets across four IE tasks, we demonstrate that current advanced LLMs consistently exhibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. Therefore, we conclude that LLMs are not effective few-shot information extractors in general. Nonetheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle challenging samples that SLMs struggle with. And moreover, we propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs serve as filters and LLMs serve as rerankers. By prompting LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements (2.4% F1-gain on average) on various IE tasks, with an acceptable time and cost investment.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

43 of 75 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017On Calibration of Modern Neural Networkscited by 4,860
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
31 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  10. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  13. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  14. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  16. 2022Unified Structure Generation for Universal Information Extractioncited by 220
  17. 2022Is GPT-3 a Good Data Annotator?cited by 208
  18. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  19. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  20. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  21. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 150
  22. 2020MAVEN: A Massive General Domain Event Detection Datasetcited by 103
  23. 2020TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Taskcited by 96
  24. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  25. 2022Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Againcited by 83
  26. 2023Exploring the Feasibility of ChatGPT for Event Extractioncited by 76
  27. 2022Code4Struct: Code Generation for Few-Shot Event Structure Predictioncited by 70
  28. 2022Recitation-Augmented Language Modelscited by 65
  29. 2022Prompt for Extraction? PAIE: Prompting Argument Interaction for Event Argument Extractioncited by 54
  30. 2022Summarization as Indirect Supervision for Relation Extractioncited by 31
  31. 2022Label Semantics for Few Shot Named Entity Recognitioncited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 48; 2025: 28; 2026: 10.

119 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  3. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  4. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  5. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  6. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  7. 2023Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detectioncited by 69
  8. 2023Self-Verification Improves Few-Shot Clinical Information Extractioncited by 52
  9. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  10. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  11. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  12. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
38 more
  1. 2025A Survey of Large Language Model Agents for Question Answeringcited by 46
  2. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  3. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  4. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  5. 2023Prompting Large Language Models for Counterfactual Generation: An Empirical Studycited by 36
  6. 2024LongHealth: A Question Answering Benchmark with Long Clinical Documentscited by 33
  7. 2024Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extractioncited by 33
  8. 2024Planning Like Human: A Dual-process Framework for Dialogue Planningcited by 32
  9. 2023How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?cited by 30
  10. 2023Revisiting Large Language Models as Zero-shot Relation Extractorscited by 29
  11. 2023A Comprehensive Evaluation of Large Language Models on Legal Judgment Predictioncited by 27
  12. 2023Primacy Effect of ChatGPTcited by 27
  13. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  14. 2024Deductive Beam Search: Decoding Deducible Rationale for Chain-of-Thought Reasoningcited by 24
  15. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  16. 2024A Survey of Conversational Searchcited by 22
  17. 2023ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extractioncited by 20
  18. 2023Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrationscited by 20
  19. 2024SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documentscited by 20
  20. 2024SFR-RAG: Towards Contextually Faithful LLMscited by 19
  21. 2023Improving Large Language Models in Event Relation Logical Predictioncited by 17
  22. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performancecited by 17
  23. 2023Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?cited by 16
  24. 2023How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domaincited by 14
  25. 2023A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomnesscited by 14
  26. 2024Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractorscited by 13
  27. 2023FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Predictioncited by 11
  28. 2024Mining experimental data from Materials Science literature with Large Language Models: an evaluation studycited by 11
  29. 2025GLiREL -- Generalist Model for Zero-Shot Relation Extractioncited by 11
  30. 2023PIVOINE: Instruction Tuning for Open-world Information Extractioncited by 10
  31. 2024Rethinking Skill Extraction in the Job Market Domain using Large Language Modelscited by 10
  32. 2023InstructIE: A Bilingual Instruction-based Information Extraction Datasetcited by 9
  33. 2023RAPL: A Relation-Aware Prototype Learning Approach for Few-Shot Document-Level Relation Extractioncited by 9
  34. 2024Grasping the Essentials: Tailoring Large Language Models for Zero-Shot Relation Extractioncited by 9
  35. 2023Make a Choice! Knowledge Base Question Answering with In-Context Learningcited by 8
  36. 2024Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answeringcited by 8
  37. 2024MockLLM: A Multi-Agent Behavior Collaboration Framework for Online Job Seeking and Recruitingcited by 8
  38. 2024Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Studycited by 8

50 most cited of 119 All 119 (JSON)