mlsanityData to 7 Oct 2026

Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks

arXiv 2305.14201 ↗First posted 23 May 2023cs.LG cs.AI cs.CLcited by 84

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Goat, a fine-tuned LLaMA model that significantly outperforms GPT-4 on a range of arithmetic tasks. Fine-tuned on a synthetically generated dataset, Goat achieves state-of-the-art performance on BIG-bench arithmetic sub-task. In particular, the zero-shot Goat-7B matches or even surpasses the accuracy achieved by the few-shot PaLM-540B. Surprisingly, Goat can achieve near-perfect accuracy on large-number addition and subtraction through supervised fine-tuning only, which is almost impossible with previous pretrained language models, such as Bloom, OPT, GPT-NeoX, etc. We attribute Goat's exceptional performance to LLaMA's consistent tokenization of numbers. To tackle more challenging tasks like large-number multiplication and division, we propose an approach that classifies tasks based on their learnability, and subsequently decomposes unlearnable tasks, such as multi-digit multiplication and division, into a series of learnable tasks by leveraging basic arithmetic principles. We thoroughly examine the performance of our model, offering a comprehensive evaluation of the effectiveness of our proposed decomposition steps. Additionally, Goat-7B can be easily trained using LoRA on a 24GB VRAM GPU, facilitating reproducibility for other researchers. We release our model, dataset, and the Python script for dataset generation.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 31 of its 40 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

31 of 40 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
19 more
  1. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  8. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  9. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  10. 2023Instruction Tuning with GPT-4cited by 676
  11. 2022PAL: Program-aided Language Modelscited by 635
  12. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  13. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  14. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  15. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  16. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  17. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  18. 2022Sub-Task Decomposition Enables Learning in Sequence to Sequence Taskscited by 41
  19. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 26; 2024: 37; 2025: 18; 2026: 3.

84 papers.

Sort:

By area:

  1. 2023A Comprehensive Overview of Large Language Modelscited by 639
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  4. 2024What matters when building vision-language models?cited by 321
  5. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  6. 2025Hierarchical Reasoning Modelcited by 167
  7. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoningcited by 117
  8. 2024Fine-Tuned Language Models Generate Stable Inorganic Materials as Textcited by 92
  9. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  10. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  11. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 65
  12. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
38 more
  1. 2024BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuningcited by 54
  2. 2024FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributionscited by 52
  3. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 51
  4. 2024Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmarkcited by 38
  5. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  6. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  7. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  8. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  9. 2024Code Pretraining Improves Entity Tracking Abilities of Language Modelscited by 31
  10. 2023Toolink: Linking Toolkit Creation and Using through Chain-of-Solving on Open-Source Modelcited by 29
  11. 2023DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversationscited by 29
  12. 2023Mathematical Language Models: A Surveycited by 24
  13. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 23
  14. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  15. 2023Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Explorationcited by 18
  16. 2023Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networkscited by 18
  17. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  18. 2023DeltaZip: Efficient Serving of Multiple Full-Model-Tuned LLMscited by 17
  19. 2024Small Language Models are Good Too: An Empirical Study of Zero-Shot Classificationcited by 17
  20. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  21. 2024Text2Chart31: Instruction Tuning for Chart Generation with Automatic Feedbackcited by 16
  22. 2024Boosting Large Language Models with Socratic Method for Conversational Mathematics Teachingcited by 14
  23. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  24. 2024Regression-aware Inference with LLMscited by 13
  25. 2023Arithmetic with Language Models: from Memorization to Computationcited by 12
  26. 2023Improving Length-Generalization in Transformers via Task Hintingcited by 12
  27. 2024CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructionscited by 12
  28. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  29. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  30. 2025BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenizationcited by 11
  31. 2024OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understandingcited by 10
  32. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 10
  33. 2023Distilling Rule-based Knowledge into Large Language Modelscited by 9
  34. 2025From Reasoning to Super-Intelligence: A Search-Theoretic Perspectivecited by 9
  35. 2023Chain-of-Thought Reasoning is a Policy Improvement Operatorcited by 8
  36. 2024Reverse That Number! Decoding Order Matters in Arithmetic Learningcited by 8
  37. 2024Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythiacited by 8
  38. 2024A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classificationcited by 8

50 most cited of 84 All 84 (JSON)