# L3DMC: Lifelong Learning using Distillation via Mixed-Curvature Space

arXiv: https://arxiv.org/abs/2307.16459 · HTML: https://mlsanity.com/paper/2307.16459/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [The Liver Tumor Segmentation Benchmark (LiTS)](https://mlsanity.com/paper/1901.04056/) (2019)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Hyperbolic Graph Convolutional Neural Networks](https://mlsanity.com/paper/1910.12933/) (2019)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [MedMNIST Classification Decathlon: A Lightweight AutoML Benchmark for Medical Image Analysis](https://mlsanity.com/paper/2010.14925/) (2020)
- [Continuous Hierarchical Representations with Poincar'e Variational Auto-Encoders](https://mlsanity.com/paper/1901.06033/) (2019)
- [Constant Curvature Graph Convolutional Networks](https://mlsanity.com/paper/1911.05076/) (2019)
- [Toward Understanding Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/1908.01091/) (2019)
- [Geodesic Exponential Kernels: When Curvature and Linearity Conflict](https://mlsanity.com/paper/1411.0296/) (2014)
- [Kernel Methods on Riemannian Manifolds with Gaussian RBF Kernels](https://mlsanity.com/paper/1412.0265/) (2014)
- [LifeLonger: A Benchmark for Continual Disease Classification](https://mlsanity.com/paper/2204.05737/) (2022)
- [InCloud: Incremental Learning for Point Cloud Place Recognition](https://mlsanity.com/paper/2203.00807/) (2022)

## Built on by (most cited first, 6 of 6)

- [M2Distill: Multi-Modal Distillation for Lifelong Imitation Learning](https://mlsanity.com/paper/2410.00064/) (2024)
- [Continual Learning in Medical Imaging: A Survey and Practical Analysis](https://mlsanity.com/paper/2405.13482/) (2024)
- [Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Images](https://mlsanity.com/paper/2504.20033/) (2025)
- [Dynamic Robot-Assisted Surgery with Hierarchical Class-Incremental Semantic Segmentation](https://mlsanity.com/paper/2508.01713/) (2025)
- [Flashbacks to Harmonize Stability and Plasticity in Continual Learning](https://mlsanity.com/paper/2506.00477/) (2025)
- [SPREAD: Subspace Representation Distillation for Lifelong Imitation Learning](https://mlsanity.com/paper/2603.08763/) (2026)

## Abstract (arXiv)

The performance of a lifelong learning (L3) model degrades when it is trained on a series of tasks, as the geometrical formation of the embedding space changes while learning novel concepts sequentially. The majority of existing L3 approaches operate on a fixed-curvature (e.g., zero-curvature Euclidean) space that is not necessarily suitable for modeling the complex geometric structure of data. Furthermore, the distillation strategies apply constraints directly on low-dimensional embeddings, discouraging the L3 model from learning new concepts by making the model highly stable. To address the problem, we propose a distillation strategy named L3DMC that operates on mixed-curvature spaces to preserve the already-learned knowledge by modeling and maintaining complex geometrical structures. We propose to embed the projected low dimensional embedding of fixed-curvature spaces (Euclidean and hyperbolic) to higher-dimensional Reproducing Kernel Hilbert Space (RKHS) using a positive-definite kernel function to attain rich representation. Afterward, we optimize the L3 model by minimizing the discrepancies between the new sample representation and the subspace constructed using the old representation in RKHS. L3DMC is capable of adapting new knowledge better without forgetting old knowledge as it combines the representation power of multiple fixed-curvature spaces and is performed on higher-dimensional RKHS. Thorough experiments on three benchmarks demonstrate the effectiveness of our proposed distillation strategy for medical image classification in L3 settings. Our code implementation is publicly available at https://github.com/csiro-robotics/L3DMC.