# Primitive Agentic First-Order Optimization

arXiv: https://arxiv.org/abs/2406.04841 · HTML: https://mlsanity.com/paper/2406.04841/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 21 of 21)

- [Cyclical Learning Rates for Training Neural Networks](https://mlsanity.com/paper/1506.01186/) (2015)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights](https://mlsanity.com/paper/1503.01243/) (2015)
- [Learning to Optimize](https://mlsanity.com/paper/1606.01885/) (2016)
- [On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice](https://mlsanity.com/paper/2007.15745/) (2020)
- [Settling the Sample Complexity of Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.05275/) (2022)
- [The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares](https://mlsanity.com/paper/1904.12838/) (2019)
- [Equilibrated adaptive learning rates for non-convex optimization](https://mlsanity.com/paper/1502.04390/) (2015)
- [Optimization-Based Control for Dynamic Legged Robots](https://mlsanity.com/paper/2211.11644/) (2022)
- [On Learning Rates and Schr"odinger Operators](https://mlsanity.com/paper/2004.06977/) (2020)
- [Learning Convex Optimization Control Policies](https://mlsanity.com/paper/1912.09529/) (2019)
- [PriorBand: Practical Hyperparameter Optimization in the Age of Deep Learning](https://mlsanity.com/paper/2306.12370/) (2023)
- [Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking](https://mlsanity.com/paper/2010.04542/) (2020)
- [Learning Simple Algorithms from Examples](https://mlsanity.com/paper/1511.07275/) (2015)
- [Selecting and Composing Learning Rate Policies for Deep Neural Networks](https://mlsanity.com/paper/2210.12936/) (2022)
- [Benchmarking for Metaheuristic Black-Box Optimization: Perspectives and Open Challenges](https://mlsanity.com/paper/2007.00541/) (2020)
- [Agent-based Collaborative Random Search for Hyper-parameter Tuning and Global Function Optimization](https://mlsanity.com/paper/2303.03394/) (2023)
- [Global optimization test problems based on random field composition](https://mlsanity.com/paper/1807.05096/) (2018)
- [Computational Rational Engineering and Development: Synergies and Opportunities](https://mlsanity.com/paper/2201.06922/) (2021)

## Abstract (arXiv)

Efficient numerical optimization methods can improve performance and reduce the environmental impact of computing in many applications. This work presents a proof-of-concept study combining primitive state representations and agent-environment interactions as first-order optimizers in the setting of budget-limited optimization. Through reinforcement learning (RL) over a set of training instances of an optimization problem class, optimal policies for sequential update selection of algorithmic iteration steps are approximated in generally formulated low-dimensional partial state representations that consider aspects of progress and resource use. For the investigated case studies, deployment of the trained agents to unseen instances of the quadratic optimization problem classes outperformed conventional optimal algorithms with optimized hyperparameters. The results show that elementary RL methods combined with succinct partial state representations can be used as heuristics to manage complexity in RL-based optimization, paving the way for agentic optimization approaches.