mlsanityData to 13 Oct 2026

FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only

arXiv 2408.01323 ↗First posted 2 Aug 2024cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Instruction fine-tuning stands as a crucial advancement in leveraging large language models (LLMs) for enhanced task performance. However, the annotation of instruction datasets has traditionally been expensive and laborious, often relying on manual annotations or costly API calls of proprietary LLMs. To address these challenges, we introduce FANNO, a fully autonomous, open-sourced framework that revolutionizes the annotation process without the need for pre-existing annotated data. Utilizing a Mistral-7b-instruct model, FANNO efficiently produces diverse and high-quality datasets through a structured process involving document pre-screening, instruction generation, and response generation. Experiments on Open LLM Leaderboard and AlpacaEval benchmark show that the FANNO can generate high-quality data with diversity and complexity for free, comparable to human-annotated or cleaned datasets like Alpaca-GPT4-Cleaned.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

28 of 57 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
16 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,091
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,063
  3. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 865
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 760
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 697
  6. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 407
  7. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 357
  8. 2018Constituency Parsing with a Self-Attentive Encodercited by 197
  9. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  10. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  11. 2018Multilingual Constituency Parsing with Self-Attention and Pre-Trainingcited by 117
  12. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  13. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
  14. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  15. 2023Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuningcited by 26
  16. 2024PlanGPT: Enhancing Urban Planning with Tailored Language Model and Efficient Retrievalcited by 8

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 4.

11 papers.

  1. 2024A Survey of Small Language Modelscited by 31
  2. 2024MdEval: Massively Multilingual Code Debuggingcited by 16
  3. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  4. 2025TAG-INSTRUCT: Controlled Instruction Complexity Enhancement through Structure-based Augmentationcited by 3
  5. 2026FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarioscited by 3
  6. 2025PlanGPT-VL: Enhancing Urban Planning with Domain-Specific Vision-Language Modelscited by 2
  7. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithmscited by 1
  8. 2026InstructDiff: Domain-Adaptive Data Selection via Differential Entropy for Efficient LLM Fine-Tuningcited by 1
  9. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptationcited by 0
  10. 2026Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQLcited by 0
  11. 2026Instruction Quality Matters: Refining Instructions for Effective Preference Learningcited by 0