mlsanityData to 7 Oct 2026

Reevaluating Meta-Learning Optimization Algorithms Through Contextual Self-Modulation

arXiv 2410.01655 ↗First posted 2 Oct 2024cs.LG math.DScited by 2

Not on the map. Why?

Abstract (arXiv)

Contextual Self-Modulation (CSM) (Nzoyem et al., 2025) is a potent regularization mechanism for Neural Context Flows (NCFs) which demonstrates powerful meta-learning on physical systems. However, CSM has limitations in its applicability across different modalities and in high-data regimes. In this work, we introduce two extensions: iCSM which expands CSM to infinite-dimensional variations by embedding the contexts into a function space, and StochasticNCF which improves scalability by providing a low-cost approximation of meta-gradient updates through a sampled set of nearest environments. These extensions are demonstrated through comprehensive experimentation on a range of tasks, including dynamical systems, computer vision challenges, and curve fitting problems. Additionally, we incorporate higher-order Taylor expansions via Taylor-Mode automatic differentiation, revealing that higher-order approximations do not necessarily enhance generalization. Finally, we demonstrate how CSM can be integrated into other meta-learning frameworks with FlashCAVIA, a computationally efficient extension of the CAVIA meta-learning framework (Zintgraf et al., 2019). Together, these contributions highlight the significant benefits of CSM and indicate that its strengths in meta-learning and out-of-distribution tasks are particularly well-suited to physical systems. Our open-source library, designed for modular integration of self-modulation into contextual meta-learning workflows, is available at https://github.com/ddrous/self-mod.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

42 of 58 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  5. 2018Neural Ordinary Differential Equationscited by 3,733
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2017Deep Setscited by 1,394
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  10. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  11. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  12. 2022Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's nextcited by 603
30 more
  1. 2019Meta-Learning with Differentiable Convex Optimizationcited by 461
  2. 2018Conditional Neural Processescited by 421
  3. 2018Meta-learning with differentiable closed-form solverscited by 395
  4. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  5. 2019Attentive Neural Processescited by 280
  6. 2022On Neural Differential Equationscited by 264
  7. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  8. 2022From data to functa: Your data point is a function and you can treat it like onecited by 183
  9. 2017Searching for Activation Functionscited by 158
  10. 2020Dissecting Neural ODEscited by 150
  11. 2019Convolutional Conditional Neural Processescited by 139
  12. 2021Neural ODE Processescited by 57
  13. 2020Improving Generalization in Meta-learning via Task Augmentationcited by 55
  14. 2021Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptationcited by 50
  15. 2022Generalizing to New Physical Systems via Context-Informed Dynamics Modelcited by 49
  16. 2021LEADS: Learning Dynamical Systems that Generalize Across Environmentscited by 39
  17. 2024Out-of-Domain Generalization in Dynamical Systems Reconstructioncited by 33
  18. 2023Autoregressive Conditional Neural Processescited by 32
  19. 2021Implicit energy regularization of neural ordinary-differential-equation controlcited by 29
  20. 2021Optimizing Reusable Knowledge for Continual Learning via Metalearningcited by 24
  21. 2017Neural Program Meta-Inductioncited by 23
  22. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  23. 2023Martingale Posterior Neural Processescited by 15
  24. 2021Context Meta-Reinforcement Learning via Neuromodulationcited by 11
  25. 2023Interpretable Meta-Learning of Physical Systemscited by 11
  26. 2022Monotone Learningcited by 9
  27. 2022Near-optimal control of dynamical systems with neural ordinary differential equationscited by 9
  28. 2024Neural Context Flows for Meta-Learning of Dynamical Systemscited by 9
  29. 2023A Comparison of Mesh-Free Differentiable Programming and Data-Driven Strategies for Optimal Control under PDE Constraintscited by 6
  30. 2021Meta-learning using privileged information for dynamicscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  2. 2025Towards Foundational Models for Dynamical System Reconstruction: Hierarchical Meta-Learning via Mixture of Expertscited by 5