mlsanityData to 7 Oct 2026

Batch Prompting: Efficient Inference with Large Language Model APIs

arXiv 2301.08721 ↗First posted 19 Jan 2023cs.CL cs.AIcited by 84

Not on the map. Why?

Abstract (arXiv)

Performing inference on large volumes of samples with large language models (LLMs) can be computationally and financially costly in industry and real-world use. We propose batch prompting, a simple yet effective prompting approach that enables the LLM to run inference in batches, instead of one sample at a time. Our method reduces both token and time costs while retaining downstream performance. We theoretically demonstrate that under a few-shot in-context learning setting, the inference costs decrease almost inverse linearly with the number of samples in each batch. We extensively validate the effectiveness of batch prompting on ten datasets across commonsense QA, arithmetic reasoning, and NLI/NLU: batch prompting significantly~(up to 5x with six samples in batch) reduces the LLM (Codex) inference token and time costs while achieving better or comparable performance. For state-of-the-art Chat-based LLMs, e.g., GPT-3.5 and GPT-4, we show the benefits of batch prompting also hold. Further analysis shows that the number of samples in each batch and the complexity of tasks affect its performance. Moreover, batch prompting can be applied across different reasoning methods using LLMs. Our code can be found at the site https://github.com/xlang-ai/batch-prompting.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 34 of its 80 references that are in the map. First posted on arXiv 19 Jan 2023.

Builds on

34 of 80 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
22 more
  1. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  6. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  8. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  9. 2022PAL: Program-aided Language Modelscited by 635
  10. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  11. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  12. 2016Solving General Arithmetic Word Problemscited by 353
  13. 2021Random Feature Attentioncited by 350
  14. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  15. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  16. 2022Binding Language Models in Symbolic Languagescited by 224
  17. 2022In-context Examples Selection for Machine Translationcited by 172
  18. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  19. 2020Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translationcited by 104
  20. 2021Finetuning Pretrained Transformers into RNNscited by 94
  21. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  22. 2021ABC: Attention with Bounded-memory Controlcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 24; 2025: 21; 2026: 19.

84 papers.

Sort:

By area:

  1. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  2. 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 174
  3. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  4. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  5. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipelinecited by 86
  6. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  7. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
  8. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoningcited by 61
  9. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  10. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  11. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  12. 2023Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environmentcited by 37
38 more
  1. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  2. 2024OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM Agentscited by 35
  3. 2023Large Language Models as Data Preprocessorscited by 34
  4. 2024Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimizationcited by 29
  5. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
  6. 2023Diversity of Thought Improves Reasoning Abilities of LLMscited by 24
  7. 2023Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneckcited by 21
  8. 2024Position: Leverage Foundational Models for Black-Box Optimizationcited by 21
  9. 2024A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinementcited by 21
  10. 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
  11. 2023Just Ask One More Time! Self-Agreement Improves Reasoning of Language Models in (Almost) All Scenarioscited by 18
  12. 2023BatchPrompt: Accomplish more with lesscited by 16
  13. 2023Cost-Effective In-Context Learning for Entity Resolution: A Design Space Explorationcited by 15
  14. 2024CigaR: Cost-efficient Program Repair with LLMscited by 14
  15. 2025Optimizing Sequential Multi-Step Tasks with Parallel LLM Agentscited by 14
  16. 2024PromptIntern: Saving Inference Costs by Internalizing Recurrent Prompt during Large Language Model Fine-tuningcited by 13
  17. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 13
  18. 2024CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulnesscited by 12
  19. 2023Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulationscited by 11
  20. 2024EcoAct: Economic Agent Determines When to Register What Actioncited by 11
  21. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
  22. 2024Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuningcited by 9
  23. 2023Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learningcited by 8
  24. 2026Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understandingcited by 8
  25. 2024MEG: Medical Knowledge-Augmented Large Language Models for Question Answeringcited by 6
  26. 2025Revisiting Prompt Optimization with Large Reasoning Models-A Case Study on Event Extractioncited by 6
  27. 2023Reducing Sequence Length by Predicting Edit Operations with Large Language Modelscited by 5
  28. 2024Towards Unified Alignment Between Agents, Humans, and Environmentcited by 5
  29. 2024Evaluating LLMs with Multiple Problems at oncecited by 5
  30. 2025Tests as Prompt: A Test-Driven-Development Benchmark for LLM Code Generationcited by 5
  31. 2025Hybrid OCR-LLM Framework for Enterprise-Scale Document Information Extraction Under Copy-heavy Taskcited by 5
  32. 2026ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressurecited by 4
  33. 2024Hybrid Querying Over Relational Databases and Large Language Modelscited by 3
  34. 2024Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Promptingcited by 3
  35. 2024Augmenting Compliance-Guaranteed Customer Service Chatbots: Context-Aware Knowledge Expansion with Large Language Modelscited by 3
  36. 2025Can LLMs Help Uncover Insights about LLMs? A Large-Scale, Evolving Literature Analysis of Frontier LLMscited by 3
  37. 2025BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compressioncited by 3
  38. 2025MAC-Tuning: LLM Multi-Compositional Problem Reasoning with Enhanced Knowledge Boundary Awarenesscited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Recursive Language Models
  2. 2025Dynamic Compressing Prompts for Efficient Inference of Large Language Models
  3. 2025PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference
  4. 2025TRPrompt: Bootstrapping Query-Aware Prompt Optimization from Textual Rewards
  5. 2023Prompt Sketching for Large Language Models
  6. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks
  7. 2025Learning a Single Token to Replace Long System Prompts in LLMs
  8. 2025PromptCoT 2.0: Scaling Prompt Synthesis for Large Language Model Reasoning