mlsanityData to 7 Oct 2026

Archon: An Architecture Search Framework for Inference-Time Techniques

arXiv 2409.15254 ↗First posted 23 Sep 2024cs.LG cs.AI cs.CLcited by 59

Not on the map. Why?

Cited by 30 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Inference-time techniques, such as repeated sampling or iterative revisions, are emerging as powerful ways to enhance large-language models (LLMs) at test time. However, best practices for developing systems that combine these techniques remain underdeveloped due to our limited understanding of the utility of each technique across models and tasks, the interactions between them, and the massive search space for combining them. To address these challenges, we introduce Archon, a modular and automated framework for optimizing the process of selecting and combining inference-time techniques and LLMs. Given a compute budget and a set of available LLMs, Archon explores a large design space to discover optimized configurations tailored to target benchmarks. It can design custom or general-purpose architectures that advance the Pareto frontier of accuracy vs. maximum token budget compared to top-performing baselines. Across instruction-following, reasoning, and coding tasks, we show that Archon can leverage additional inference compute budget to design systems that outperform frontier models such as OpenAI's o1, GPT-4o, and Claude 3.5 Sonnet by an average of 15.1%.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 39 of its 74 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

39 of 74 references.

References · 39

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2021Program Synthesis with Large Language Modelscited by 3,509
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
27 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  10. 2023Large Language Modelscited by 774
  11. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  12. 2017Progressive Neural Architecture Searchcited by 483
  13. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  14. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  15. 2024Automated Design of Agentic Systemscited by 360
  16. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 253
  17. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  18. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  19. 2024More Agents Is All You Needcited by 174
  20. 2020A Survey on Evolutionary Neural Architecture Searchcited by 142
  21. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 137
  22. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  23. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  24. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  25. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 104
  26. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 73
  27. 2024Networks of Networks: Complexity Class Principles Applied to Compound AI Systems Designcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 37; 2026: 18.

59 papers.

Sort:

By area:

  1. 2024AFlow: Automating Agentic Workflow Generationcited by 395
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2025Multi-agent Architecture Search via Agentic Supernetcited by 164
  4. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  5. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  6. 2025S*: Test Time Scaling for Code Generationcited by 95
  7. 2025FlowReasoner: Reinforcing Query-Level Meta-Agentscited by 56
  8. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Modelscited by 56
  9. 2025Atom of Thoughts for Markov LLM Test-Time Scalingcited by 54
  10. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  11. 2025Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Modelscited by 51
  12. 2025ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimizationcited by 50
38 more
  1. 2025EvoFlow: Evolving Diverse Agentic Workflows On The Flycited by 49
  2. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
  3. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  4. 2025Self-Supervised Prompt Optimizationcited by 37
  5. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  6. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  7. 2025Internet of Agents: Fundamentals, Applications, and Challengescited by 31
  8. 2025Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflowscited by 23
  9. 2026Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokenscited by 21
  10. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  11. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  12. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 17
  13. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  14. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  15. 2025A2Flow: Automating Agentic Workflow Generation via Self-Adaptive Abstraction Operatorscited by 14
  16. 2025How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoningcited by 13
  17. 2025Abacus: A Cost-Based Optimizer for Semantic Operator Systemscited by 11
  18. 2025DebFlow: Automating Agent Creation via Agent Debatecited by 10
  19. 2025An Information Theoretic Perspective on Agentic System Designcited by 8
  20. 2025Helpful Agent Meets Deceptive Judge: Understanding Vulnerabilities in Agentic Workflowscited by 6
  21. 2025GenoMAS: A Multi-Agent Framework for Scientific Discovery via Code-Driven Gene Expression Analysiscited by 6
  22. 2025Slm-mux: Orchestrating small language models for reasoningcited by 5
  23. 2025Inefficiencies of Meta Agents for Agent Designcited by 5
  24. 2026When to Trust the Cheap Check: Weak and Strong Verification for Reasoningcited by 5
  25. 2025Introspection of Thought Helps AI Agentscited by 4
  26. 2025Trade in Minutes! Rationality-Driven Agentic System for Quantitative Financial Tradingcited by 4
  27. 2025Generalizing Test-time Compute-optimal Scaling as an Optimizable Graphcited by 4
  28. 2026Automated Discovery Has No Universally Superior Harnesscited by 4
  29. 2025Searching Meta Reasoning Skeleton to Guide LLM Reasoningcited by 3
  30. 2026EVOCHAMBER: Test-Time Co-evolution of Multi-Agent System at Individual, Team, and Population Scalescited by 3
  31. 2026BayesFlow: A Probability Inference Framework for Meta-Agent Assisted Workflow Generationcited by 2
  32. 2026Agentic Systems as Boosting Weak Reasoning Modelscited by 2
  33. 2025LangProBe: a Language Programs Benchmarkcited by 1
  34. 2025COSMOS: Predictable and Cost-Effective Adaptation of LLMscited by 1
  35. 2025Polymath: A Self-Optimizing Agent with Dynamic Hierarchical Workflowcited by 1
  36. 2025FutureWeaver: Planning Test-Time Compute for Multi-Agent Systems with Modularized Collaborationcited by 1
  37. 2026AgentSlimming: Towards Efficient and Cost-Aware Multi-Agent Systemscited by 1
  38. 2026AoA: Theorem Proving Agent over Abstract Syntax Tree of Redesigned Languagecited by 1

50 most cited of 59 All 59 (JSON)