mlsanityData to 7 Oct 2026

In-Context Semi-Supervised Learning

arXiv 2512.15934 ↗First posted 17 Dec 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

There has been significant recent interest in understanding the capacity of Transformers for in-context learning (ICL), yet most theory focuses on supervised settings with explicitly labeled pairs. In practice, Transformers often perform well even when labels are sparse or absent, suggesting crucial structure within unlabeled contextual demonstrations. We introduce and study in-context semi-supervised learning (IC-SSL), where a small set of labeled examples is accompanied by many unlabeled points, and show that Transformers can leverage the unlabeled context to learn a robust, context-dependent representation. This representation enables accurate predictions and markedly improves performance in low-label regimes, offering foundational insights into how Transformers exploit unlabeled context for representation learning within the ICL framework.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 17 Dec 2025.

Builds on

32 of 52 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
20 more
  1. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  2. 2022Transformers learn in-context by gradient descentcited by 714
  3. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  4. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  5. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  6. 2021MetaICL: Learning to Learn In Contextcited by 450
  7. 2024The Platonic Representation Hypothesiscited by 362
  8. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  9. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  10. 2022Torsional Diffusion for Molecular Conformer Generationcited by 262
  11. 2022Riemannian Score-Based Generative Modellingcited by 216
  12. 2024Many-Shot In-Context Learningcited by 203
  13. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  14. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  15. 2021Learning disentangled representations via product manifold projectioncited by 20
  16. 2020Product Manifold Learningcited by 16
  17. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  18. 2025When and How Unlabeled Data Provably Improve In-Context Learningcited by 5
  19. 2024On Understanding Attention-Based In-Context Learning for Categorical Datacited by 3
  20. 2024Graph Transformers Dream of Electric Flowcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classificationcited by 0