mlsanityData to 14 Oct 2026

Uncovering Scaling Laws for Large Language Models via Inverse Problems

arXiv 2509.07909 ↗First posted 9 Sep 2025cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are large-scale pretrained models that have achieved remarkable success across diverse domains. These successes have been driven by unprecedented complexity and scale in both data and computations. However, due to the high costs of training such models, brute-force trial-and-error approaches to improve LLMs are not feasible. Inspired by the success of inverse problems in uncovering fundamental scientific laws, this position paper advocates that inverse problems can also efficiently uncover scaling laws that guide the building of LLMs to achieve the desirable performance with significantly better cost-effectiveness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 73 of its 106 references that are in the map. First posted on arXiv 9 Sep 2025.

Builds on

73 of 106 references.

References · 73

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  12. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  13. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  14. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  15. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  16. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  17. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  18. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  19. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  20. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  21. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  22. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  23. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  24. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  25. 2024A Survey on Data Selection for Language Modelscited by 236
  26. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 215
  27. 2023Continual Pre-training of Language Modelscited by 198
  28. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  29. 2018Model Evaluation, Model Selection, and Algorithm Selection in Machine Learningcited by 157
  30. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  31. 2020Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functionscited by 144
  32. 2024Towards Safer Large Language Models through Machine Unlearningcited by 139
  33. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
  34. 2023CodeGen2: Lessons for Training LLMs on Programming and Natural Languagescited by 125
  35. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  36. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  37. 2024What makes unlearning hard and what to do about itcited by 89
  38. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 85

50 most cited of 73 All 73 (JSON)

Built on by

Not cited yet.