# Reevaluating Meta-Learning Optimization Algorithms Through Contextual Self-Modulation

arXiv: https://arxiv.org/abs/2410.01655 · HTML: https://mlsanity.com/paper/2410.01655/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 30 of 42)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [On First-Order Meta-Learning Algorithms](https://mlsanity.com/paper/1803.02999/) (2018)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Conditional Neural Processes](https://mlsanity.com/paper/1807.01613/) (2018)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Attentive Neural Processes](https://mlsanity.com/paper/1901.05761/) (2019)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Adaptive Gradient-Based Meta-Learning Methods](https://mlsanity.com/paper/1906.02717/) (2019)
- [Searching for Activation Functions](https://mlsanity.com/paper/1710.05941/) (2017)
- [Dissecting Neural ODEs](https://mlsanity.com/paper/2002.08071/) (2020)
- [Convolutional Conditional Neural Processes](https://mlsanity.com/paper/1910.13556/) (2019)
- [Neural ODE Processes](https://mlsanity.com/paper/2103.12413/) (2021)
- [Improving Generalization in Meta-learning via Task Augmentation](https://mlsanity.com/paper/2007.13040/) (2020)
- [Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation](https://mlsanity.com/paper/2106.09017/) (2021)
- [Generalizing to New Physical Systems via Context-Informed Dynamics Model](https://mlsanity.com/paper/2202.01889/) (2022)
- [LEADS: Learning Dynamical Systems that Generalize Across Environments](https://mlsanity.com/paper/2106.04546/) (2021)
- [Autoregressive Conditional Neural Processes](https://mlsanity.com/paper/2303.14468/) (2023)
- [Out-of-Domain Generalization in Dynamical Systems Reconstruction](https://mlsanity.com/paper/2402.18377/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications](https://mlsanity.com/paper/2503.07137/) (2025)
- [Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Experts](https://mlsanity.com/paper/2502.05335/) (2025)

## Abstract (arXiv)

Contextual Self-Modulation (CSM) (Nzoyem et al., 2025) is a potent regularization mechanism for Neural Context Flows (NCFs) which demonstrates powerful meta-learning on physical systems. However, CSM has limitations in its applicability across different modalities and in high-data regimes. In this work, we introduce two extensions: iCSM which expands CSM to infinite-dimensional variations by embedding the contexts into a function space, and StochasticNCF which improves scalability by providing a low-cost approximation of meta-gradient updates through a sampled set of nearest environments. These extensions are demonstrated through comprehensive experimentation on a range of tasks, including dynamical systems, computer vision challenges, and curve fitting problems. Additionally, we incorporate higher-order Taylor expansions via Taylor-Mode automatic differentiation, revealing that higher-order approximations do not necessarily enhance generalization. Finally, we demonstrate how CSM can be integrated into other meta-learning frameworks with FlashCAVIA, a computationally efficient extension of the CAVIA meta-learning framework (Zintgraf et al., 2019). Together, these contributions highlight the significant benefits of CSM and indicate that its strengths in meta-learning and out-of-distribution tasks are particularly well-suited to physical systems. Our open-source library, designed for modular integration of self-modulation into contextual meta-learning workflows, is available at https://github.com/ddrous/self-mod.