mlsanityData to 14 Oct 2026

Concept-aware Training Improves In-context Learning Ability of Language Models

arXiv 2305.13775 ↗First posted 23 May 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Many recent language models (LMs) of Transformers family exhibit so-called in-context learning (ICL) ability, manifested in the LMs' ability to modulate their function by a task described in a natural language input. Previous work curating these models assumes that ICL emerges from vast over-parametrization or the scale of multi-task training. However, a complementary branch of recent theoretical work attributes ICL emergence to specific properties of training data and creates functional in-context learners in small-scale, synthetic settings. Inspired by recent findings on data properties driving the emergence of ICL, we propose a method to create LMs able to better utilize the in-context information, by constructing training scenarios where it is beneficial for the LM to capture the analogical reasoning concepts. We measure that data sampling of Concept-aware Training (CoAT) consistently improves models' reasoning ability. As a result, the in-context learners trained with CoAT on only two datasets of a single (QA) task perform comparably to larger models trained on 1600+ tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 34 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

28 of 34 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  12. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
16 more
  1. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  2. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  4. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2021MetaICL: Learning to Learn In Contextcited by 454
  7. 2023Larger language models do in-context learning differentlycited by 394
  8. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  9. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  10. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  11. 2023The Learnability of In-Context Learningcited by 127
  12. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  13. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 79
  14. 2022Adaptor: Objective-Centric Adaptation Framework for Language Modelscited by 7
  15. 2022Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contextscited by 7
  16. 2023Resources and Few-shot Learners for In-context Learning in Slavic Languagescited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Concept-aware Data Construction Improves In-context Learning of Language Models
  2. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  3. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
  4. 2024In-Context Language Learning: Architectures and Algorithms
  5. 2024Revisiting In-context Learning Inference Circuit in Large Language Models
  6. 2024On the Role of Model Prior in Real-World Inductive Reasoning
  7. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS
  8. 2023TART: A plug-and-play Transformer module for task-agnostic reasoning