# In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks

arXiv: https://arxiv.org/abs/2410.01548 · HTML: https://mlsanity.com/paper/2410.01548/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2205.06743/) (2022)
- [Fine-tuned Language Models are Continual Learners](https://mlsanity.com/paper/2205.12393/) (2022)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [Theoretical Analysis of Domain Adaptation with Optimal Transport](https://mlsanity.com/paper/1610.04420/) (2016)
- [Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluations](https://mlsanity.com/paper/2306.04618/) (2023)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator](https://mlsanity.com/paper/2206.08082/) (2022)
- [InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions](https://mlsanity.com/paper/2403.11435/) (2024)
- [A Review of Deep Transfer Learning and Recent Advancements](https://mlsanity.com/paper/2201.09679/) (2022)
- [Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations](https://mlsanity.com/paper/2305.15035/) (2023)
- [Unlocking Continual Learning Abilities in Language Models](https://mlsanity.com/paper/2406.17245/) (2024)
- [Selective Demonstrations for Cross-domain Text-to-SQL](https://mlsanity.com/paper/2310.06302/) (2023)
- [Auto-ICL: In-Context Learning without Human Supervision](https://mlsanity.com/paper/2311.09263/) (2023)
- [ConPET: Continual Parameter-Efficient Tuning for Large Language Models](https://mlsanity.com/paper/2309.14763/) (2023)
- [Demonstration Augmentation for Zero-shot In-context Learning](https://mlsanity.com/paper/2406.01224/) (2024)
- [Improving Demonstration Diversity by Human-Free Fusing for Text-to-SQL](https://mlsanity.com/paper/2402.10663/) (2024)
- [Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughts](https://mlsanity.com/paper/2402.10691/) (2024)
- [Continual Learning via Manifold Expansion Replay](https://mlsanity.com/paper/2310.08038/) (2023)

## Built on by (most cited first, 3 of 3)

- [Why Search When You Can Transfer? Amortized Agentic Workflow Design from Structural Priors](https://mlsanity.com/paper/2604.25012/) (2026)
- [AutoPR: Let's Automate Your Academic Promotion!](https://mlsanity.com/paper/2510.09558/) (2025)
- [CrossICL: Cross-Task In-Context Learning via Unsupervised Demonstration Transfer](https://mlsanity.com/paper/2505.24143/) (2025)

## Abstract (arXiv)

In-context learning (ICL) is an effective approach to help large language models (LLMs) adapt to various tasks by providing demonstrations of the target task. Considering the high cost of labeling demonstrations, many methods propose synthesizing demonstrations from scratch using LLMs. However, the quality of the demonstrations synthesized from scratch is limited by the capabilities and knowledge of LLMs. To address this, inspired by transfer learning, we propose In-Context Transfer Learning (ICTL), which synthesizes target task demonstrations by transferring labeled demonstrations from similar source tasks. ICTL consists of two steps: source sampling and target transfer. First, we define an optimization objective, which minimizes transfer error to sample source demonstrations similar to the target task. Then, we employ LLMs to transfer the sampled source demonstrations to the target task, matching the definition and format of the target task. Experiments on Super-NI show that ICTL outperforms synthesis from scratch by 2.0% on average, demonstrating the effectiveness of our method.