# Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuning

arXiv: https://arxiv.org/abs/2307.02053 · HTML: https://mlsanity.com/paper/2307.02053/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 16 references that are in the map.
- First posted on arXiv 5 Jul 2023.

## Builds on (most cited first, 14 of 14)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Full Parameter Fine-tuning for Large Language Models with Limited Resources](https://mlsanity.com/paper/2306.09782/) (2023)
- [INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models](https://mlsanity.com/paper/2306.04757/) (2023)

## Built on by (most cited first, 18 of 18)

- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models](https://mlsanity.com/paper/2402.05935/) (2024)
- [MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation](https://mlsanity.com/paper/2308.08239/) (2023)
- [OWL: A Large Language Model for IT Operations](https://mlsanity.com/paper/2309.09298/) (2023)
- [Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful](https://mlsanity.com/paper/2507.07101/) (2025)
- [Aligning Large Language Models with Human: A Survey](https://mlsanity.com/paper/2307.12966/) (2023)
- [Falcon-UI: Understanding GUI Before Following User Instructions](https://mlsanity.com/paper/2412.09362/) (2024)
- [Instruction Tuning with Human Curriculum](https://mlsanity.com/paper/2310.09518/) (2023)
- [Abstraction-of-Thought Makes Language Models Better Reasoners](https://mlsanity.com/paper/2406.12442/) (2024)
- [Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability](https://mlsanity.com/paper/2506.15629/) (2025)
- [Xiwu: A Basis Flexible and Learnable LLM for High Energy Physics](https://mlsanity.com/paper/2404.08001/) (2024)
- [Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates](https://mlsanity.com/paper/2408.12263/) (2024)
- [FIPO: Free-form Instruction-oriented Prompt Optimization with Preference Dataset and Modular Fine-tuning Schema](https://mlsanity.com/paper/2402.11811/) (2024)
- [Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths](https://mlsanity.com/paper/2410.10858/) (2024)
- [The Right Time Matters: Data Arrangement Affects Zero-Shot Generalization in Instruction Tuning](https://mlsanity.com/paper/2406.11721/) (2024)
- [DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents](https://mlsanity.com/paper/2503.11170/) (2025)
- [DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning](https://mlsanity.com/paper/2603.12932/) (2026)
- [Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models](https://mlsanity.com/paper/2410.16163/) (2024)

## Abstract (arXiv)

Recently, the release of INSTRUCTEVAL has provided valuable insights into the performance of large language models (LLMs) that utilize encoder-decoder or decoder-only architecture. Interestingly, despite being introduced four years ago, T5-based LLMs, such as FLAN-T5, continue to outperform the latest decoder-based LLMs, such as LLAMA and VICUNA, on tasks that require general problem-solving skills. This performance discrepancy can be attributed to three key factors: (1) Pre-training data, (2) Backbone architecture, and (3) Instruction dataset. In this technical report, our main focus is on investigating the impact of the third factor by leveraging VICUNA, a large language model based on LLAMA, which has undergone fine-tuning on ChatGPT conversations. To achieve this objective, we fine-tuned VICUNA using a customized instruction dataset collection called FLANMINI. This collection includes a subset of the large-scale instruction dataset known as FLAN, as well as various code-related datasets and conversational datasets derived from ChatGPT/GPT-4. This dataset comprises a large number of tasks that demand problem-solving skills. Our experimental findings strongly indicate that the enhanced problem-solving abilities of our model, FLACUNA, are obtained through fine-tuning VICUNA on the FLAN dataset, leading to significant improvements across numerous benchmark datasets in INSTRUCTEVAL. FLACUNA is publicly available at https://huggingface.co/declare-lab/flacuna-13b-v1.0.