# In-Context Guidance: Learning Inter-Task Synergies via Numerical Foundational Models for Few-Shot Multitask Optimization

arXiv: https://arxiv.org/abs/2609.25836 · HTML: https://mlsanity.com/paper/2609.25836/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 43 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 18 of 18)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [A Tutorial on Bayesian Optimization](https://mlsanity.com/paper/1807.02811/) (2018)
- [GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration](https://mlsanity.com/paper/1809.11165/) (2018)
- [Transformers Can Do Bayesian Inference](https://mlsanity.com/paper/2112.10510/) (2021)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [Unexpected Improvements to Expected Improvement for Bayesian Optimization](https://mlsanity.com/paper/2310.20708/) (2023)
- [TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models](https://mlsanity.com/paper/2511.08667/) (2025)
- [TabDPT: Scaling Tabular Foundation Models on Real Data](https://mlsanity.com/paper/2410.18164/) (2024)
- [Quality Diversity for Multi-task Optimization](https://mlsanity.com/paper/2003.04407/) (2020)
- [Bayesian Optimization with LLM-Based Acquisition Functions for Natural Language Preference Elicitation](https://mlsanity.com/paper/2405.00981/) (2024)
- [Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed](https://mlsanity.com/paper/2203.09179/) (2022)
- [Evolutionary Multitasking for Single-objective Continuous Optimization: Benchmark Problems, Performance Metric, and Baseline Results](https://mlsanity.com/paper/1706.03470/) (2017)
- [Scalable Transfer Evolutionary Optimization: Coping with Big Task Instances](https://mlsanity.com/paper/2012.01830/) (2020)
- [(θl, θu)-Parametric Multi-Task Optimization: Joint Search in Solution and Infinite Task Spaces](https://mlsanity.com/paper/2503.08394/) (2025)
- [Surrogate-Assisted Search with Competitive Knowledge Transfer for Expensive Optimization](https://mlsanity.com/paper/2408.07176/) (2024)
- [Large Language Model as Meta-Surrogate for Data-Driven Many-Task Optimization: A Proof-of-Principle Study](https://mlsanity.com/paper/2503.08301/) (2025)
- [Multi-Task Surrogate-Assisted Search with Bayesian Competitive Knowledge Transfer for Expensive Optimization](https://mlsanity.com/paper/2510.23407/) (2025)
- [Amortized Multi-Objective Optimization Across Tasks with Generative Solution Modeling](https://mlsanity.com/paper/2511.09598/) (2025)

## Abstract (arXiv)

Multi-task optimization (MTO) addresses a set of optimization tasks simultaneously, often suffering from inaccurate inter-task relationship estimation under limited evaluation budgets, leading to negative transfer. This paper introduces In-Context Guidance Multitask Optimization (ICG-MTO), a novel framework that leverages numerical foundational models to improve inter-task coupling estimation in few-shot scenarios. Unlike conventional methods that rely solely on scarce observed data, ICG-MTO employs a frozen foundational model to infer auxiliary guidance through in-context learning. The framework operates through three stages: constructing an algorithm-specific in-context query from evaluated solutions, using the foundational model to infer a guidance signal characterizing predictive relationships among tasks, and translating this signal into algorithm-specific guidance for maximum-a-posteriori coupling estimation. This approach provides regularization during the early, data-scarce stages of optimization and gradually relinquishes control as task-specific observations accumulate. We instantiate the framework in multitask Bayesian optimization as ICG-MTBO, using directional fitness-class queries to guide inter-task coupling estimation, and further instantiate it in MFEA-II using decision-space-overlap queries to guide random mating probability estimation. Experiments across synthetic benchmarks and a real-world robot arm control problem, together with evaluations under different acquisition functions and evolutionary multitasking, demonstrate the effectiveness and generality of ICG-MTO for few-shot multitask optimization.