# Contrasting Exploration in Parameter and Action Space: A Zeroth-Order Optimization Perspective

arXiv: https://arxiv.org/abs/1901.11503 · HTML: https://mlsanity.com/paper/1901.11503/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Jan 2019.

## Built on by (most cited first, 27 of 27)

- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training](https://mlsanity.com/paper/2310.02025/) (2023)
- [Fast and Efficient Locomotion via Learned Gait Transitions](https://mlsanity.com/paper/2104.04644/) (2021)
- [Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning](https://mlsanity.com/paper/2402.15751/) (2024)
- [Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design](https://mlsanity.com/paper/2110.03659/) (2021)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Average-reward model-free reinforcement learning: a systematic review and literature mapping](https://mlsanity.com/paper/2010.08920/) (2020)
- [Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning](https://mlsanity.com/paper/2502.03304/) (2025)
- [Derivative-Free Reinforcement Learning: A Review](https://mlsanity.com/paper/2102.05710/) (2021)
- [Symmetry-Aware Robot Design with Structured Subgroups](https://mlsanity.com/paper/2306.00036/) (2023)
- [Distributed Multi-Agent Reinforcement Learning Based on Graph-Induced Local Value Functions](https://mlsanity.com/paper/2202.13046/) (2022)
- [Fully Zeroth-Order Bilevel Programming via Gaussian Smoothing](https://mlsanity.com/paper/2404.00158/) (2024)
- [Assessing Quality-Diversity Neuro-Evolution Algorithms Performance in Hard Exploration Problems](https://mlsanity.com/paper/2211.13742/) (2022)
- [Learning One Abstract Bit at a Time Through Self-Invented Experiments Encoded as Neural Networks](https://mlsanity.com/paper/2212.14374/) (2022)
- [Rendering Wireless Environments Useful for Gradient Estimators: A Zero-Order Stochastic Federated Learning Method](https://mlsanity.com/paper/2401.17460/) (2024)
- [Task2Morph: Differentiable Task-inspired Framework for Contact-Aware Robot Design](https://mlsanity.com/paper/2403.19093/) (2024)
- [End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost](https://mlsanity.com/paper/2509.00031/) (2025)
- [Distributed Cooperative Multi-Agent Reinforcement Learning with Directed Coordination Graph](https://mlsanity.com/paper/2201.04962/) (2022)
- [Zero-Order One-Point Estimate with Distributed Stochastic Gradient-Tracking Technique](https://mlsanity.com/paper/2210.05618/) (2022)
- [Forward Learning for Gradient-based Black-box Saliency Map Generation](https://mlsanity.com/paper/2403.15603/) (2024)
- [Black-Box Generalization: Stability of Zeroth-Order Learning](https://mlsanity.com/paper/2202.06880/) (2022)
- [Learning Sampling Policy for Faster Derivative Free Optimization](https://mlsanity.com/paper/2104.04405/) (2021)
- [Model-Free Learning of Optimal Deterministic Resource Allocations in Wireless Systems via Action-Space Exploration](https://mlsanity.com/paper/2108.10352/) (2021)
- [Adaptive Policy Backbone via Shared Network](https://mlsanity.com/paper/2509.22310/) (2025)
- [Bayesian Optimization for Dynamic Pricing and Learning](https://mlsanity.com/paper/2510.12447/) (2025)
- [ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Training](https://mlsanity.com/paper/2603.13115/) (2026)
- [StructZO: Structured Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning](https://mlsanity.com/paper/2610.16451/) (2026)

## Abstract (arXiv)

Black-box optimizers that explore in parameter space have often been shown to outperform more sophisticated action space exploration methods developed specifically for the reinforcement learning problem. We examine these black-box methods closely to identify situations in which they are worse than action space exploration methods and those in which they are superior. Through simple theoretical analyses, we prove that complexity of exploration in parameter space depends on the dimensionality of parameter space, while complexity of exploration in action space depends on both the dimensionality of action space and horizon length. This is also demonstrated empirically by comparing simple exploration methods on several model problems, including Contextual Bandit, Linear Regression and Reinforcement Learning in continuous control.