mlsanityData to 14 Oct 2026

Context Tuning for In-Context Optimization

arXiv 2507.04221 ↗First posted 6 Jul 2025cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of large language models (LLMs) without weight updates. In-Context Learning (ICL) forms a memory representation of the demonstrations in a single forward pass but cannot refine it when insufficient. Prompt-based methods offer lightweight adaptation by optimizing a trainable prompt or prefix but initialize it independently of the demonstrations. In contrast, Context Tuning leverages the model's inherent ICL ability to initialize a trainable memory representation from demonstrations, then refines it through gradient-based optimization. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms both ICL and traditional prompt-based adaptation methods while achieving competitive accuracy with Test-Time Training at significantly higher training efficiency.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 53 of its 64 references that are in the map. First posted on arXiv 6 Jul 2025.

Builds on

53 of 64 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2025Qwen3 Technical Reportcited by 8,820
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
38 more
  1. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  11. 2021GPT Understands, Toocited by 716
  12. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  13. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  14. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  15. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  16. 2021MetaICL: Learning to Learn In Contextcited by 454
  17. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  18. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  19. 2022Test-Time Training with Masked Autoencoderscited by 212
  20. 2024Many-Shot In-Context Learningcited by 206
  21. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  22. 2021Meta-learning via Language Model In-context Tuningcited by 164
  23. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  24. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  25. 2023End-to-End Diffusion Latent Optimization Improves Classifier Guidancecited by 108
  26. 2023Finding Support Examples for In-Context Learningcited by 104
  27. 2024ARC Prize 2024: Technical Reportcited by 104
  28. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  29. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 96
  30. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  31. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 86
  32. 2023In-context Learning and Gradient Descent Revisitedcited by 20
  33. 2024Searching Latent Program Spacescited by 20
  34. 2025Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspectivecited by 17
  35. 2023Self-supervised Meta-Prompt Learning with Meta-Gradient Regularization for Few-shot Generalizationcited by 15
  36. 2023In-Context Exemplars as Clues to Retrieving from Large Associative Memorycited by 13
  37. 2024ProCreate, Don't Reproduce! Propulsive Energy Diffusion for Creative Generationcited by 13
  38. 2023Learning to Initialize: Can Meta Learning Improve Cross-task Generalization in Prompt Tuning?cited by 11

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  2. 2025Fine-Tuned In-Context Learners for Efficient Adaptationcited by 2
  3. 2026HyperThink: Text-to-Parameter Hypernetworks for Efficient Reasoningcited by 1