mlsanityData to 7 Oct 2026

SeCoKD: Aligning Large Language Models for In-Context Learning with Fewer Shots

arXiv 2406.14208 ↗First posted 20 Jun 2024cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Previous studies have shown that demonstrations can significantly help Large Language Models (LLMs ) perform better on the given tasks. However, this so-called In-Context Learning ( ICL ) ability is very sensitive to the presenting context, and often dozens of demonstrations are needed. In this work, we investigate if we can reduce the shot number while still maintaining a competitive performance. We present SeCoKD, a self-Knowledge Distillation ( KD ) training framework that aligns the student model with a heavily prompted variation, thereby increasing the utilization of a single demonstration. We experiment with the SeCoKD across three LLMs and six benchmarks focusing mainly on reasoning tasks. Results show that our method outperforms the base model and Supervised Fine-tuning ( SFT ), especially in zero-shot and one-shot settings by 30% and 10%, respectively. Moreover, SeCoKD brings little negative artifacts when evaluated on new tasks, which is more robust than Supervised Fine-tuning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 40 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

33 of 40 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
20 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  2. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  3. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  4. 2016Sequence-Level Knowledge Distillationcited by 761
  5. 2022A Survey on In-context Learningcited by 717
  6. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  8. 2021MetaICL: Learning to Learn In Contextcited by 450
  9. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  10. 2023Larger language models do in-context learning differentlycited by 386
  11. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  12. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221
  13. 2022Active Example Selection for In-Context Learningcited by 195
  14. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  15. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  16. 2023Unified Demonstration Retriever for In-Context Learningcited by 141
  17. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  18. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  19. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  20. 2023ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extractioncited by 38

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Models
  2. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  3. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
  4. 2025Improving In-Context Learning with Reasoning Distillation
  5. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator
  6. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
  7. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learning
  8. 2024Concept-aware Data Construction Improves In-context Learning of Language Models