mlsanityData to 7 Oct 2026

Reconciling In-Context and In-Weight Learning via Dual Representation Space Encoding

arXiv 2603.13459 ↗First posted 13 Mar 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) is a valuable capability exhibited by Transformers pretrained on diverse sequence tasks. However, previous studies have observed that ICL often conflicts with the model's inherent in-weight learning (IWL) ability. By examining the representation space learned by a toy model in synthetic experiments, we identify the shared encoding space for context and samples in Transformers as a potential source of this conflict. To address this, we modify the model architecture to separately encode the context and samples into two distinct spaces: a task representation space and a sample representation space. We model these two spaces under a simple yet principled framework, assuming a linear representational structure and treating them as a pair of dual spaces. Both theoretical analysis and empirical results demonstrate the effectiveness of our proposed architecture, CoQE, in the single-value answer setting. It not only enhances ICL performance through improved representation learning, but also successfully reconciles ICL and IWL capabilities across synthetic few-shot classification and a newly designed pseudo-arithmetic task. Code: https://github.com/McGuinnessChen/dual-representation-space-encoding

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

52 of 62 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2022Planning-oriented Autonomous Drivingcited by 1,068
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  12. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
38 more
  1. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  2. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  3. 2023Language Models Represent Space and Timecited by 306
  4. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  5. 2023In-Context Learning Creates Task Vectorscited by 302
  6. 2023Function Vectors in Large Language Modelscited by 292
  7. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
  8. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  9. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  10. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 112
  11. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  12. 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 101
  13. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  14. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
  15. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  16. 2023In-Context Learning through the Bayesian Prismcited by 78
  17. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  18. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  19. 2022Discovering Latent Concepts Learned in BERTcited by 56
  20. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 52
  21. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 48
  22. 2024Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformerscited by 46
  23. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  24. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  25. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  26. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  27. 2023A Closer Look at In-Context Learning under Distribution Shiftscited by 25
  28. 2025Task Vectors in In-Context Learning: Emergence, Formation, and Benefitcited by 20
  29. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  30. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 19
  31. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  32. 2024Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgettingcited by 15
  33. 2024Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superpositioncited by 13
  34. 2024Does learning the right latent variables necessarily improve in-context learning?cited by 12
  35. 2024Feature contamination: Neural networks learn uncorrelated features and fail to generalizecited by 11
  36. 2024Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspectivecited by 9
  37. 2024Transformers Handle Endogeneity in In-Context Linear Regressioncited by 8
  38. 2025When Do Neural Networks Learn World Models?cited by 8

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.