# OrCo: Towards Better Generalization via Orthogonality and Contrast for Few-Shot Class-Incremental Learning

arXiv: https://arxiv.org/abs/2403.18550 · HTML: https://mlsanity.com/paper/2403.18550/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 52 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Maintaining Discrimination and Fairness in Class Incremental Learning](https://mlsanity.com/paper/1911.07053/) (2019)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)
- [Few-Shot Incremental Learning with Continually Evolved Classifiers](https://mlsanity.com/paper/2104.03047/) (2021)
- [FearNet: Brain-Inspired Model for Incremental Learning](https://mlsanity.com/paper/1711.10563/) (2017)
- [Continual Prototype Evolution: Learning Online from Non-Stationary Data Streams](https://mlsanity.com/paper/2009.00919/) (2020)
- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)
- [Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting](https://mlsanity.com/paper/1802.02950/) (2018)
- [Forward Compatible Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2203.06953/) (2022)

## Built on by (most cited first, 26 of 26)

- [Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model](https://mlsanity.com/paper/2503.04543/) (2025)
- [Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2407.06136/) (2024)
- [Towards Few-Shot Learning in the Open World: A Review and Beyond](https://mlsanity.com/paper/2408.09722/) (2024)
- [Latest Advancements Towards Catastrophic Forgetting under Data Scarcity: A Comprehensive Survey on Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2502.08181/) (2025)
- [Learn by Reasoning: Analogical Weight Generation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2503.21258/) (2025)
- [DevFD: Developmental Face Forgery Detection by Learning Shared and Orthogonal LoRA Subspaces](https://mlsanity.com/paper/2509.19230/) (2025)
- [Unsupervised Few-Shot Continual Learning for Remote Sensing Image Scene Classification](https://mlsanity.com/paper/2406.18574/) (2024)
- [Continuous Knowledge-Preserving Decomposition with Adaptive Layer Selection for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2501.05017/) (2025)
- [Controllable Forgetting Mechanism for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2501.15998/) (2025)
- [Beyond Synthetic Replays: Turning Diffusion Features into Few-Shot Class-Incremental Learning Knowledge](https://mlsanity.com/paper/2503.23402/) (2025)
- [Cross-Branch Orthogonality for Improved Generalization in Face Deepfake Detection](https://mlsanity.com/paper/2505.04888/) (2025)
- [Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2506.19558/) (2025)
- [Few-shot Class-Incremental Learning via Generative Co-Memory Regularization](https://mlsanity.com/paper/2601.07117/) (2026)
- [Embedding Space Allocation with Angle-Norm Joint Classifiers for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2411.09250/) (2024)
- [Fresh-CL: Feature Realignment through Experts on Hypersphere in Continual Learning](https://mlsanity.com/paper/2501.02198/) (2025)
- [Site-Level Fine-Tuning with Progressive Layer Freezing: Towards Robust Prediction of Bronchopulmonary Dysplasia from Day-1 Chest Radiographs in Extremely Preterm Infants](https://mlsanity.com/paper/2507.12269/) (2025)
- [MoTiC: Momentum Tightness and Contrast for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2509.19664/) (2025)
- [Diffusion-Classifier Synergy: Reward-Aligned Learning via Mutual Boosting Loop for FSCIL](https://mlsanity.com/paper/2510.03608/) (2025)
- [Breaking Forgetting: Training-Free Few-Shot Class-Incremental Learning via Conditional Diffusion](https://mlsanity.com/paper/2511.18516/) (2025)
- [Automatic Attack Discovery for Few-Shot Class-Incremental Learning via Large Language Models](https://mlsanity.com/paper/2512.03882/) (2025)
- [Open Horizons: Evaluating Deep Models in the Wild](https://mlsanity.com/paper/2512.12146/) (2025)
- [CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2601.08519/) (2026)
- [CASP: Few-Shot Class-Incremental Learning with CLS Token Attention Steering Prompts](https://mlsanity.com/paper/2601.16773/) (2026)
- [When Sensing Varies with Contexts: Context Probing for Tactile Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2603.25115/) (2026)
- [Optical-Guided Neural Collapse for SAR Few-Shot Class Incremental Learning](https://mlsanity.com/paper/2606.04528/) (2026)
- [Selective Backpropagation for Efficient Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2610.04003/) (2026)

## Abstract (arXiv)

Few-Shot Class-Incremental Learning (FSCIL) introduces a paradigm in which the problem space expands with limited data. FSCIL methods inherently face the challenge of catastrophic forgetting as data arrives incrementally, making models susceptible to overwriting previously acquired knowledge. Moreover, given the scarcity of labeled samples available at any given time, models may be prone to overfitting and find it challenging to strike a balance between extensive pretraining and the limited incremental data. To address these challenges, we propose the OrCo framework built on two core principles: features' orthogonality in the representation space, and contrastive learning. In particular, we improve the generalization of the embedding space by employing a combination of supervised and self-supervised contrastive losses during the pretraining phase. Additionally, we introduce OrCo loss to address challenges arising from data limitations during incremental sessions. Through feature space perturbations and orthogonality between classes, the OrCo loss maximizes margins and reserves space for the following incremental data. This, in turn, ensures the accommodation of incoming classes in the feature space without compromising previously acquired knowledge. Our experimental results showcase state-of-the-art performance across three benchmark datasets, including mini-ImageNet, CIFAR100, and CUB datasets. Code is available at https://github.com/noorahmedds/OrCo