Data to 14 Oct 2026

Small Models are Valuable Plug-ins for Large Language Models

arXiv 2305.08848 ↗First posted 15 May 2023cs.CL cs.AI cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) such as GPT-3 and GPT-4 are powerful but their weights are often publicly unavailable and their immense sizes make the models difficult to be tuned with common hardware. As a result, effectively tuning these models with large-scale supervised data can be challenging. As an alternative, In-Context Learning (ICL) can only use a small number of supervised examples due to context length limits. In this paper, we propose Super In-Context Learning (SuperICL) which allows black-box LLMs to work with locally fine-tuned smaller models, resulting in superior performance on supervised tasks. Our experiments demonstrate that SuperICL can improve performance beyond state-of-the-art fine-tuned models while addressing the instability problem of in-context learning. Furthermore, SuperICL can enhance the capabilities of smaller models, such as multilinguality and interpretability.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

33 of 40 references.

References · 33

2020Language Models are Few-Shot Learnerscited by 32,419
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
2015Distilling the Knowledge in a Neural Networkcited by 10,324
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
2016Understanding deep learning requires rethinking generalizationcited by 1,674
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190

21 more

2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
2022Transformers learn in-context by gradient descentcited by 732
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
2021MetaICL: Learning to Learn In Contextcited by 454
2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
2022Active Example Selection for In-Context Learningcited by 196
2022In-context Examples Selection for Machine Translationcited by 172
2023Compositional Exemplars for In-context Learningcited by 144
2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 139
2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
2022Can Large Language Models Truly Understand Prompts? A Case Study with Negated Promptscited by 59
2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
2023OpenICL: An Open-Source Framework for In-context Learningcited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 15; 2025: 9; 2026: 4.

42 papers.

Sort:

By area:

2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
2023Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agentscited by 77
2023Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detectioncited by 72
2023Active Learning Principles for In-Context Learning with Large Language Modelscited by 54
2024Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Taskscited by 48
2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 36
2023Distilled GPT for Source Code Summarizationcited by 31
2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 29
2025The Truth Becomes Clearer Through Debate! Multi-Agent Systems with Large Language Models Unmask Fake Newscited by 29
2024TC-RAG:Turing-Complete RAG's Case study on Medical LLM Systemscited by 28
2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
2023Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Productioncited by 21

30 more

2023Supervised Knowledge Makes Large Language Models Better In-context Learnerscited by 20
2024Online Cascade Learning for Efficient Inference over Streamscited by 20
2025Collab-RAG: Boosting Retrieval-Augmented Generation for Complex Question Answering via White-Box and Black-Box LLM Collaborationcited by 20
2025In-Context Meta LoRA Generationcited by 15
2024LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertaintycited by 12
2024Towards Sustainable Large Language Model Servingcited by 12
2024Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoningcited by 10
2024GPT for Games: An Updated Scoping Review (2020-2024)cited by 10
2024Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMscited by 9
2025CLIPGaussian: Universal and Multimodal Style Transfer Based on Gaussian Splattingcited by 8
2023MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examplescited by 7
2023Plug-and-Play Document Modules for Pre-trained Modelscited by 6
2023Plug-and-Play Knowledge Injection for Pre-trained Language Modelscited by 6
2024Improving In-Context Learning with Small Language Model Ensemblescited by 4
2023Data Distribution Bottlenecks in Grounding Language Models to Knowledge Basescited by 3
2023SCALE: Synergized Collaboration of Asymmetric Language Translation Enginescited by 3
2024AICL: Action In-Context Learning for Video Diffusion Modelcited by 2
2026Interfaze: The Future of AI is built on Task-Specific Small Modelscited by 2
2023Insert-expansions for Tool-enabled Conversational Agentscited by 1
2024Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AIcited by 1
2025HERA: Hybrid Edge-cloud Resource Allocation for Cost-Efficient AI Agentscited by 1
2025KCM: KAN-Based Collaboration Models Enhance Pretrained Large Modelscited by 1
2025Effective and Efficient Threat Hunting with Small Language Modelscited by 1
2023One-stop Training of Multiple Capacity Modelscited by 0
2024Mitigating the Negative Impact of Over-association for Conversational Query Productioncited by 0
2025LeForecast: Enterprise Hybrid Forecast by Time Series Intelligencecited by 0
2025Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integrationcited by 0
2026DeEscalWild: A Real-World Benchmark for Automated De-Escalation Training with SLMscited by 0
2026MedMIX: Modality-Internal Expert Fusion for Multimodal Medical Diagnosiscited by 0
2026SCOPE: A Lightweight-training LLM Framework for Air Traffic Control Readback Monitoringcited by 0