# Escaping Local Optima in the Waddington Landscape: A Two-Stage TRPO-PPO Approach for Single-Cell Perturbation Analysis

arXiv: https://arxiv.org/abs/2510.13018 · HTML: https://mlsanity.com/paper/2510.13018/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 40 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 7 of 7)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Predicting Cellular Responses to Novel Drug Perturbations at a Single-Cell Resolution](https://mlsanity.com/paper/2204.13545/) (2022)
- [Trust Region-Guided Proximal Policy Optimization](https://mlsanity.com/paper/1901.10314/) (2019)
- [First-Order Preconditioning via Hypergradient Descent](https://mlsanity.com/paper/1910.08461/) (2019)
- [PPO in the Fisher-Rao geometry](https://mlsanity.com/paper/2506.03757/) (2025)
- [Clipped-Objective Policy Gradients for Pessimistic Policy Optimization](https://mlsanity.com/paper/2311.05846/) (2023)

## Abstract (arXiv)

Modeling cellular responses to genetic and chemical perturbations remains a central challenge in single-cell biology. Existing data-driven frameworks have advanced perturbation prediction through variational autoencoders, chemically conditioned autoencoders, and large-scale transformer pretraining. However, most existing models rely exclusively on either in silico perturbation data or experimental perturbation data but rarely integrate both, limiting their ability to generalize and validate predictions across simulated and real biological contexts in a digital twin system. Moreover, the models are prone to local optima in the nonconvex Waddington landscape of cell fate decisions, where poor initialization can trap trajectories in spurious lineages. In this work, we introduce a two-stage reinforcement learning algorithm for modeling single-cell perturbation. We first compute an explicit natural gradient update using Fisher-vector products and a conjugate gradient solver, scaled by a KL trust-region constraint to provide a safe, curvature-aware first step for the policy. Starting with these preconditioned parameters, we then apply a second phase of proximal policy optimization (PPO) with a KL penalty, exploiting minibatch efficiency to refine the policy. We demonstrate that this initialization strategy substantially improves generalization on Single-cell RNA sequencing (scRNA-seq) perturbation analysis in a digital twin system.