LangProBe: a Language Programs Benchmark
Not on the map. Why?
Abstract (arXiv)
Composing language models (LMs) into multi-step language programs and automatically optimizing their modular prompts is now a mainstream paradigm for building AI systems, but the tradeoffs in this space have only scarcely been studied before. We introduce LangProBe, the first large-scale benchmark for evaluating the architectures and optimization strategies for language programs, with over 2000 combinations of tasks, architectures, optimizers, and choices of LMs. Using LangProBe, we are the first to study the impact of program architectures and optimizers (and their compositions together and with different models) on tradeoffs of quality and cost. We find that optimized language programs offer strong cost--quality Pareto improvement over raw calls to models, but simultaneously demonstrate that human judgment (or empirical decisions) about which compositions to pursue is still necessary for best performance. We will open source the code and evaluation data for LangProBe.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 27 Feb 2025.
Builds on
22 of 44 references.
References · 22
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
10 more
- 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 318
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 291
- 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 201
- 2021Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrievalcited by 63
- 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 60
- 2024The Limits of Inference Scaling Through Resamplingcited by 53
- 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
- 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 40
- 2023ScoNe: Benchmarking Negation Reasoning in Language Models With Fine-Tuning and In-Context Learningcited by 18
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.