# OpenICL: An Open-Source Framework for In-context Learning

arXiv: https://arxiv.org/abs/2303.02913 · HTML: https://mlsanity.com/paper/2303.02913/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 58 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 41)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)

## Built on by (most cited first, 30 of 51)

- [OS-Copilot: Towards Generalist Computer Agents with Self-Improvement](https://mlsanity.com/paper/2402.07456/) (2024)
- [Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis](https://mlsanity.com/paper/2304.04675/) (2023)
- [Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration](https://mlsanity.com/paper/2310.00280/) (2023)
- [Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely](https://mlsanity.com/paper/2409.14924/) (2024)
- [Revisiting Demonstration Selection Strategies in In-Context Learning](https://mlsanity.com/paper/2401.12087/) (2024)
- [Small Models are Valuable Plug-ins for Large Language Models](https://mlsanity.com/paper/2305.08848/) (2023)
- [Forward-Backward Reasoning in Large Language Models for Mathematical Verification](https://mlsanity.com/paper/2308.07758/) (2023)
- [Multilingual Machine Translation with Open Large Language Models at Practical Scale: An Empirical Study](https://mlsanity.com/paper/2502.02481/) (2025)
- [Assessment of Multimodal Large Language Models in Alignment with Human Values](https://mlsanity.com/paper/2403.17830/) (2024)
- [Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMs](https://mlsanity.com/paper/2405.02774/) (2024)
- [Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability](https://mlsanity.com/paper/2306.06688/) (2023)
- [Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning](https://mlsanity.com/paper/2312.08901/) (2023)
- [Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning](https://mlsanity.com/paper/2410.07074/) (2024)
- [LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking](https://mlsanity.com/paper/2308.04945/) (2023)
- [Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning](https://mlsanity.com/paper/2402.00667/) (2024)
- [In-Context Demonstration Selection with Cross Entropy Difference](https://mlsanity.com/paper/2305.14726/) (2023)
- [On the Noise Robustness of In-Context Learning for Text Generation](https://mlsanity.com/paper/2405.17264/) (2024)
- [Debiasing Multimodal Sarcasm Detection with Contrastive Learning](https://mlsanity.com/paper/2312.10493/) (2023)
- [The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis](https://mlsanity.com/paper/2402.12976/) (2024)
- [LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews](https://mlsanity.com/paper/2504.11042/) (2025)
- [ICLEval: Evaluating In-Context Learning Ability of Large Language Models](https://mlsanity.com/paper/2406.14955/) (2024)
- [CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelines](https://mlsanity.com/paper/2407.12797/) (2024)
- [Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions](https://mlsanity.com/paper/2409.14572/) (2024)
- [Jailbreaking? One Step Is Enough!](https://mlsanity.com/paper/2412.12621/) (2024)
- [MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning](https://mlsanity.com/paper/2305.12692/) (2023)
- [Assessing Translation capabilities of Large Language Models involving English and Indian Languages](https://mlsanity.com/paper/2311.09216/) (2023)
- [EMO: Earth Mover Distance Optimization for Auto-Regressive Language Modeling](https://mlsanity.com/paper/2310.04691/) (2023)
- [Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning](https://mlsanity.com/paper/2309.14771/) (2023)
- [A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs](https://mlsanity.com/paper/2406.17377/) (2024)
- [OpenCompass: A Universal Evaluation Platform for Large Language Models](https://mlsanity.com/paper/2605.19276/) (2026)

## Abstract (arXiv)

In recent years, In-context Learning (ICL) has gained increasing attention and emerged as the new paradigm for large language model (LLM) evaluation. Unlike traditional fine-tuning methods, ICL instead adapts the pre-trained models to unseen tasks without any parameter updates. However, the implementation of ICL is sophisticated due to the diverse retrieval and inference methods involved, as well as the varying pre-processing requirements for different models, datasets, and tasks. A unified and flexible framework for ICL is urgently needed to ease the implementation of the aforementioned components. To facilitate ICL research, we introduce OpenICL, an open-source toolkit for ICL and LLM evaluation. OpenICL is research-friendly with a highly flexible architecture that users can easily combine different components to suit their needs. It also provides various state-of-the-art retrieval and inference methods to streamline the process of adapting ICL to cutting-edge research. The effectiveness of OpenICL has been validated on a wide range of NLP tasks, including classification, QA, machine translation, and semantic parsing. As a side-product, we found OpenICL to be an efficient yet robust tool for LLMs evaluation. OpenICL is released at https://github.com/Shark-NLP/OpenICL