mlsanityData to 14 Oct 2026

Visual Program Distillation with Template-Based Augmentation

arXiv 2412.08564 ↗First posted 11 Dec 2024cs.CV cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Adapting visual programming or prompting large language models (LLMs) to generate executable code for visual tasks like visual question answering (VQA) for specialized tasks or domains remains challenging due to high annotation and inference costs. We propose a low-cost visual program distillation method that can be used for models with at most 1 billion parameters and requires no human-generated program annotations. We achieve this through synthetic data augmentation based on decoupling programs into higher-level skills, called templates, and their corresponding arguments. Experimental results show that, with a relatively small amount of question/answer data, small language models can generate high-quality specialized visual programs with the added benefit of much faster inference

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 37 of its 70 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

37 of 70 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
25 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  5. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  7. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  10. 2015Neural Module Networkscited by 376
  11. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  12. 2021Knowledge distillation: A good teacher is patient and consistentcited by 259
  13. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 243
  14. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 153
  15. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  16. 2023CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsetscited by 129
  17. 2017Learning to Reason: End-to-End Module Networks for Visual Question Answeringcited by 125
  18. 2018Generating Informative and Diverse Conversational Responses via Adversarial Information Maximizationcited by 116
  19. 2016Learning to Compose Neural Networks for Question Answeringcited by 85
  20. 2023Modular Visual Question Answering via Code Generationcited by 48
  21. 2022Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Codecited by 39
  22. 2023PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuningcited by 29
  23. 2024Enhancing Code Generation Performance of Smaller Models by Distilling the Reasoning Ability of LLMscited by 17
  24. 2024Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmerscited by 14
  25. 2024Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcementcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Neuro-Symbolic Drive: Rule-Grounded Faithful Reasoning for Driving VLAscited by 1
  2. 2026RVLM: Recursive Vision-Language Models with Adaptive Depthcited by 0
  3. 2026LLM-based Visual Code Completion for Aerospace Geometric Designcited by 0