# FlatManifold: Robust Continual Learning under Severe Label Noise and Domain Shifts via Intrinsic Manifold Flattening

arXiv: https://arxiv.org/abs/2607.05201 · HTML: https://mlsanity.com/paper/2607.05201/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 6 Jul 2026.

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research](https://mlsanity.com/paper/2308.13149/) (2023)
- [Small Language Models Need Strong Verifiers to Self-Correct Reasoning](https://mlsanity.com/paper/2404.17140/) (2024)
- [Step-level Value Preference Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2406.10858/) (2024)
- [Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents](https://mlsanity.com/paper/2412.00821/) (2024)
- [Step-KTO: Optimizing Mathematical Reasoning through Stepwise Binary Feedback](https://mlsanity.com/paper/2501.10799/) (2025)
- [Using Large Language Model to Solve and Explain Physics Word Problems Approaching Human Level](https://mlsanity.com/paper/2309.08182/) (2023)
- [Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning](https://mlsanity.com/paper/2502.14356/) (2025)
- [Evaluating the Factuality of Large Language Models using Large-Scale Knowledge Graphs](https://mlsanity.com/paper/2404.00942/) (2024)
- [Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback](https://mlsanity.com/paper/2412.06827/) (2024)

## Abstract (arXiv)

In non-stationary streaming environments, simultaneously adapting to complex, non-linear domain shifts via continual learning while mitigating the catastrophic effects of severe, uncalibrated label noise poses a fundamental mathematical challenge. In this paper, we propose FlatManifold, a novel, streamlined robust continual learning framework that utilizes a Nystr"om manifold flattening map based on the kernel trick and projection onto an orthogonalized Reproducing Kernel Hilbert Space (RKHS). Unlike traditional methods that rely on complex, error-prone sample-filtering pipelines, the proposed approach exploits the intrinsic mathematical robustness of the flattened space itself. By mapping feature distributions onto a fixed orthogonal target topology with a ridge regularizer, the framework naturally smoothes and counteracts the influence of extreme label noise during the optimization process. Concurrently, catastrophic forgetting is prevented via a continual topology brake term that leverages the covariance matrix of past experiences. Extensive evaluation on real-world multi-session robotics datasets demonstrates that even under severe conditions featuring 40% symmetric label noise, FlatManifold successfully mitigates gradient corruption. Under extreme cross-session domain shifts spanning various seasons and lighting conditions, the proposed framework establishes high generalization capabilities, significantly outperforming standard sequential optimization baselines and proving that structural linearization itself serves as a powerful mathematical barrier against distributed label corruption.