mlsanityData to 7 Oct 2026

Towards Efficient Large Language Models for Scientific Text: A Review

arXiv 2408.10729 ↗First posted 20 Aug 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have ushered in a new era for processing complex information in various fields, including science. The increasing amount of scientific literature allows these models to acquire and understand scientific knowledge effectively, thus improving their performance in a wide range of tasks. Due to the power of LLMs, they require extremely expensive computational resources, intense amounts of data, and training time. Therefore, in recent years, researchers have proposed various methodologies to make scientific LLMs more affordable. The most well-known approaches align in two directions. It can be either focusing on the size of the models or enhancing the quality of data. To date, a comprehensive review of these two families of methods has not yet been undertaken. In this paper, we (I) summarize the current advances in the emerging abilities of LLMs into more accessible AI solutions for science, and (II) investigate the challenges and opportunities of developing affordable solutions for scientific domains using LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 20 Aug 2024.

Builds on

40 of 55 references.

References · 40

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2022Galactica: A Large Language Model for Sciencecited by 736
  5. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  7. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  8. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  9. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 282
  10. 2023HuatuoGPT, towards Taming Language Model to Be a Doctorcited by 211
  11. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  12. 2023DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Taskcited by 130
28 more
  1. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  2. 2023BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinecited by 99
  3. 2023DISC-MedLLM: Bridging General Large Language Models and Real-World Medical Consultationcited by 89
  4. 2023AlpaCare:Instruction-tuned Large Language Models for Medical Applicationcited by 88
  5. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  6. 2023PMC-LLaMA: Towards Building Open-source Language Models for Medicinecited by 87
  7. 2024BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Textcited by 85
  8. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  9. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  10. 2023K2: A Foundation Language Model for Geoscience Knowledge Understanding and Utilizationcited by 69
  11. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  12. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  13. 2023OceanGPT: A Large Language Model for Ocean Science Taskscited by 42
  14. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  15. 2023Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domaincited by 38
  16. 2024SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Modelscited by 38
  17. 2024Complete and Efficient Graph Transformers for Crystal Material Property Predictioncited by 34
  18. 2023Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extractioncited by 32
  19. 2023CITB: A Benchmark for Continual Instruction Tuningcited by 32
  20. 2023Large Language Models as Master Key: Unlocking the Secrets of Materials Science with GPTcited by 29
  21. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  22. 2023MarineGPT: Unlocking Secrets of Ocean to the Publiccited by 23
  23. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMscited by 22
  24. 2023Connectivity Optimized Nested Graph Networks for Crystal Structurescited by 11
  25. 2023Continual Learning with Dirichlet Generative-based Rehearsalcited by 7
  26. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  27. 2024Selective Annotation via Data Allocation: These Data Should Be Triaged to Experts for Annotation Rather Than the Modelcited by 6
  28. 2024ADA-GNN: Atom-Distance-Angle Graph Neural Network for Crystal Material Property Predictioncited by 1

Built on by

Not cited yet.