mlsanityData to 7 Oct 2026

Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs

arXiv 2410.20749 ↗First posted 28 Oct 2024cs.LG cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Despite the impressive generative abilities of black-box large language models (LLMs), their inherent opacity hinders further advancements in capabilities such as reasoning, planning, and personalization. Existing works aim to enhance LLM capabilities via domain-specific adaptation, which require additional training on accessible model parameters, an infeasible option for black-box LLMs. To address this challenge, we introduce Matryoshka Pilot (M-Pilot), a lightweight white-box LLM controller that guides a large-scale black-box LLM generator by decomposing complex tasks into a series of intermediate outputs. Specifically, we consider the black-box LLM as an environment, with M-Pilot serving as a policy to provide intermediate guidance through prompts for driving the black-box LLM. M-Pilot is trained to pivot the outputs of the black-box LLM aligning with preferences during iterative interaction, which enables controllable multi-turn generation and self-improvement in optimizing intermediate guidance. Empirical evaluations on diverse tasks demonstrate that our method effectively enhances the capabilities of black-box LLMs in complex, long-horizon tasks. Our code is publicly available at: https://github.com/lichangh20/Matryoshka.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 52 of its 63 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

52 of 63 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
38 more
  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  8. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  9. 2023Teaching Large Language Models to Self-Debugcited by 963
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  13. 2023Large Language Models as Optimizerscited by 830
  14. 2022PAL: Program-aided Language Modelscited by 635
  15. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  16. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  17. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  18. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  19. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  20. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  21. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  22. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  23. 2023AdaPlanner: Adaptive Planning from Feedback with Language Modelscited by 193
  24. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  25. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  26. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 156
  27. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  28. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 87
  29. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  30. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  31. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  32. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 59
  33. 2023Small Models are Valuable Plug-ins for Large Language Modelscited by 42
  34. 2024In-Context Principle Learning from Mistakescited by 42
  35. 2023Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Modelscited by 40
  36. 2024MedAdapter: Efficient Test-Time Adaptation of Large Language Models towards Medical Reasoningcited by 35
  37. 2024Few-shot Personalization of LLMs with Mis-aligned Responsescited by 30
  38. 2024EPO: Hierarchical LLM Agents with Environment Preference Optimizationcited by 28

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  2. 2025How to Train Your Advisor: Steering Black-Box LLMs with Advisor Modelscited by 20
  3. 2025WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learningcited by 7
  4. 2025Hephaestus: Improving Fundamental Agent Capabilities of Large Language Models through Continual Pre-Trainingcited by 6
  5. 2026Synthetic Interaction Data for Scalable Personalization in Large Language Modelscited by 4
  6. 2026Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineeringcited by 2
  7. 2026ExecTune: Effective Steering of Black-Box LLMs with Guide Modelscited by 0
  8. 2026Exploration-Driven Optimization for Test-Time Large Language Model Reasoningcited by 0
  9. 2026AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillationcited by 0