# Damped Gauss Newton Search for Multi Metric Hyperparameter Optimization

arXiv: https://arxiv.org/abs/2401.03580 · HTML: https://mlsanity.com/paper/2401.03580/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 19 references that are in the map.
- First posted on arXiv 7 Jan 2024.

## Builds on (most cited first, 8 of 8)

- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization](https://mlsanity.com/paper/2109.09831/) (2021)
- [Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges](https://mlsanity.com/paper/2107.05847/) (2021)
- [Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview](https://mlsanity.com/paper/2206.07438/) (2022)
- [A survey on multi-objective hyperparameter optimization algorithms for Machine Learning](https://mlsanity.com/paper/2111.13755/) (2021)

## Abstract (arXiv)

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a black-box, target-seeking damped Gauss--Newton method for multiple validation responses. The method estimates trajectory-based numerical sensitivities from successive changes in hyperparameters and validation performance. These full-vector observations construct an iterative secant approximation of the local sensitivity, and a damped Gauss--Newton system jointly updates all hyperparameters. The method does not differentiate through model training or require coordinate-wise perturbation runs: after two initial evaluations establish the first secant, each subsequent iteration requires one new full-vector evaluation. Tikhonov regularization stabilizes the underdetermined local inverse problem. Unlike multi-objective HPO that approximates a Pareto front, our formulation seeks a specified operating point in multi-response performance space. We evaluate the method on four-dimensional XGBoost HPO across three classification datasets, six-dimensional LoRA/SFT post-training of Qwen3-VL-8B on a binary VQAv2 subset, and eight-dimensional threshold optimization on fixed classifier outputs. XGBoost results are comparable to grid, random, and TPE search with fewer model evaluations. The VLM study reveals non-monotonic response and sensitivity trajectories, while the threshold study demonstrates target seeking in an underdetermined system and sensitivity to initialization. Overall, the results support trajectory-based secant sensitivity as an evaluation-efficient local optimization mechanism complementary to global black-box HPO.