Foundation Model Engineering: Engineering Foundation Models Just as Engineering Software
Not on the map. Why?
Abstract (arXiv)
By treating data and models as the source code, Foundation Models (FMs) become a new type of software. Mirroring the concept of software crisis, the increasing complexity of FMs making FM crisis a tangible concern in the coming decade, appealing for new theories and methodologies from the field of software engineering. In this paper, we outline our vision of introducing Foundation Model (FM) engineering, a strategic response to the anticipated FM crisis with principled engineering methodologies. FM engineering aims to mitigate potential issues in FM development and application through the introduction of declarative, automated, and unified programming interfaces for both data and model management, reducing the complexities involved in working with FMs by providing a more structured and intuitive process for developers. Through the establishment of FM engineering, we aim to provide a robust, automated, and extensible framework that addresses the imminent challenges, and discovering new research opportunities for the software engineering field.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 50 of its 86 references that are in the map. First posted on arXiv 11 Jul 2024.
Builds on
50 of 86 references.
References · 50
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2020Knowledge Distillation: A Surveycited by 1,578
- 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
- 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
- 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
- 2021Florence: A New Foundation Model for Computer Visioncited by 802
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
- 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2021Merging Models with Fisher-Weighted Averagingcited by 613
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
- 2018Rethinking the Value of Network Pruningcited by 595
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
- 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
- 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
- 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
- 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
- 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
- 2018Model compression via distillation and quantizationcited by 287
- 2021Training Neural Networks with Fixed Sparse Maskscited by 212
- 2018Physical Adversarial Examples for Object Detectorscited by 163
- 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
- 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
- 2022KronA: Parameter Efficient Tuning with Kronecker Adaptercited by 130
- 2022Fusing finetuned models for better pretrainingcited by 110
- 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
- 2023Soft Merging of Experts with Adaptive Routingcited by 88
- 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
- 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
- 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
- 2017HoloClean: Holistic Data Repairs with Probabilistic Inferencecited by 79
- 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
- 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
- 2023LLM-based Medical Assistant Personalization with Short- and Long-Term Memory Coordinationcited by 46
- 2022Efficient Fine-Tuning of BERT Models on the Edgecited by 29
- 2020Picket: Guarding Against Corrupted Data in Tabular Data during Learning and Inferencecited by 11
Built on by
Citing papers per year: 2025: 1.
1 paper.