# C-ICL: Contrastive In-context Learning for Information Extraction

arXiv: https://arxiv.org/abs/2402.11254 · HTML: https://mlsanity.com/paper/2402.11254/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 60 references that are in the map.
- First posted on arXiv 17 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction](https://mlsanity.com/paper/1808.09602/) (2018)
- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [A Unified MRC Framework for Named Entity Recognition](https://mlsanity.com/paper/1910.11476/) (2019)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction](https://mlsanity.com/paper/2304.08085/) (2023)
- [Named Entity Recognition as Dependency Parsing](https://mlsanity.com/paper/2005.07150/) (2020)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness](https://mlsanity.com/paper/2304.11633/) (2023)
- [GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction](https://mlsanity.com/paper/2310.03668/) (2023)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [Code4Struct: Code Generation for Few-Shot Event Structure Prediction](https://mlsanity.com/paper/2210.12810/) (2022)
- [Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations](https://mlsanity.com/paper/2212.09865/) (2022)
- [CodeIE: Large Code Generation Models are Better Few-Shot Information Extractors](https://mlsanity.com/paper/2305.05711/) (2023)
- [In-context Example Selection with Influences](https://mlsanity.com/paper/2302.11042/) (2023)
- [Empirical Study of Zero-Shot NER with ChatGPT](https://mlsanity.com/paper/2310.10035/) (2023)
- [OWL: A Large Language Model for IT Operations](https://mlsanity.com/paper/2309.09298/) (2023)
- [Retrieval-Augmented Code Generation for Universal Information Extraction](https://mlsanity.com/paper/2311.02962/) (2023)
- [Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task](https://mlsanity.com/paper/2111.02086/) (2021)

## Built on by (most cited first, 23 of 23)

- [Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion](https://mlsanity.com/paper/2502.00245/) (2025)
- [Unlocking the Power of LLM Uncertainty for Active In-Context Example Selection](https://mlsanity.com/paper/2408.09172/) (2024)
- [LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking](https://mlsanity.com/paper/2404.05624/) (2024)
- [MR-UIE: Multi-Perspective Reasoning with Reinforcement Learning for Universal Information Extraction](https://mlsanity.com/paper/2509.09082/) (2025)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [GuideX: Guided Synthetic Data Generation for Zero-Shot Information Extraction](https://mlsanity.com/paper/2506.00649/) (2025)
- [H2HTalk: Evaluating Large Language Models as Emotional Companion](https://mlsanity.com/paper/2507.03543/) (2025)
- [Learning Wisdom from Errors: Promoting LLM's Continual Relation Learning through Exploiting Error Cases](https://mlsanity.com/paper/2508.12031/) (2025)
- [Document-Level Event Extraction with Definition-Driven ICL](https://mlsanity.com/paper/2408.05566/) (2024)
- [SkillVerse : Assessing and Enhancing LLMs with Tree Evaluation](https://mlsanity.com/paper/2506.00319/) (2025)
- [EMRModel: A Large Language Model for Extracting Medical Consultation Dialogues into Structured Medical Records](https://mlsanity.com/paper/2504.16448/) (2025)
- [A Multi-Agent LLM Framework for Multi-Domain Low-Resource In-Context NER via Knowledge Retrieval, Disambiguation and Reflective Analysis](https://mlsanity.com/paper/2511.19083/) (2025)
- [Structured Semantic Information Helps Retrieve Better Examples for In-Context Learning Applied to Few-Shot Relation Extraction](https://mlsanity.com/paper/2601.20803/) (2026)
- [BCL: Bayesian In-Context Learning Framework for Information Extraction](https://mlsanity.com/paper/2606.18620/) (2026)
- [LA-RL: Label-Aware Self-Reflection for Reinforcement Learning in Information Extraction](https://mlsanity.com/paper/2607.23420/) (2026)
- [Exploring Imbalanced Annotations for Effective In-Context Learning](https://mlsanity.com/paper/2502.04037/) (2025)
- [Enhancing Contrastive Demonstration Selection with Semantic Diversity for Robust In-Context Machine Translation](https://mlsanity.com/paper/2504.09305/) (2025)
- [A Training-Free Regeneration Paradigm: Contrastive Reflection Memory Guided Self-Verification and Self-Improvement](https://mlsanity.com/paper/2603.20441/) (2026)
- [LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction](https://mlsanity.com/paper/2606.29407/) (2026)
- [Two-Stage Prompt Optimization for Few-Shot Relation Extraction: From Reasoning-Guided Search to Gradient-Guided Refinement](https://mlsanity.com/paper/2606.29639/) (2026)
- [DIRECT: Direct Decoding for Efficient and Aligned Sequence Labeling with Large Language Models](https://mlsanity.com/paper/2607.26891/) (2026)
- [Query-Only Backdoor Attacks on Self-Evolving Skills via Trajectory Poisoning](https://mlsanity.com/paper/2608.08303/) (2026)
- [The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errors](https://mlsanity.com/paper/2610.16977/) (2026)

## Abstract (arXiv)

There has been increasing interest in exploring the capabilities of advanced large language models (LLMs) in the field of information extraction (IE), specifically focusing on tasks related to named entity recognition (NER) and relation extraction (RE). Although researchers are exploring the use of few-shot information extraction through in-context learning with LLMs, they tend to focus only on using correct or positive examples for demonstration, neglecting the potential value of incorporating incorrect or negative examples into the learning process. In this paper, we present c-ICL, a novel few-shot technique that leverages both correct and incorrect sample constructions to create in-context learning demonstrations. This approach enhances the ability of LLMs to extract entities and relations by utilizing prompts that incorporate not only the positive samples but also the reasoning behind them. This method allows for the identification and correction of potential interface errors. Specifically, our proposed method taps into the inherent contextual information and valuable information in hard negative samples and the nearest positive neighbors to the test and then applies the in-context learning demonstrations based on LLMs. Our experiments on various datasets indicate that c-ICL outperforms previous few-shot in-context learning methods, delivering substantial enhancements in performance across a broad spectrum of related tasks. These improvements are noteworthy, showcasing the versatility of our approach in miscellaneous scenarios.