# Retrieval-Augmented Fine-Tuning With Preference Optimization For Visual Program Generation

arXiv: https://arxiv.org/abs/2502.16529 · HTML: https://mlsanity.com/paper/2502.16529/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RAG for code, one of 15 directions in Retrieval-augmented generation.
- Builds on 23 of its 57 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Placed in

- Retrieval-augmented generation / [RAG for code](https://mlsanity.com/rag/rag-for-code/)
- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [RAFT: Adapting Language Model to Domain Specific RAG](https://mlsanity.com/paper/2403.10131/) (2024)
- [On Softmax Direct Preference Optimization for Recommendation](https://mlsanity.com/paper/2406.09215/) (2024)
- [CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences](https://mlsanity.com/paper/2403.09032/) (2024)
- [ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems](https://mlsanity.com/paper/2409.01392/) (2024)
- [West-of-N: Synthetic Preferences for Self-Improving Reward Models](https://mlsanity.com/paper/2401.12086/) (2024)
- [Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agents](https://mlsanity.com/paper/2410.14209/) (2024)
- [Does RLHF Scale? Exploring the Impacts From Data, Model, and Method](https://mlsanity.com/paper/2412.06000/) (2024)
- [InfiniteWorld: A Unified Scalable Simulation Framework for General Visual-Language Robot Interaction](https://mlsanity.com/paper/2412.05789/) (2024)
- [ChatGPT for PLC/DCS Control Logic Generation](https://mlsanity.com/paper/2305.15809/) (2023)
- [Duplicated Code Pattern Mining in Visual Programming Languages](https://mlsanity.com/paper/2107.07212/) (2021)
- [Exploring LLM Support for Generating IEC 61131-3 Graphic Language Programs](https://mlsanity.com/paper/2410.15200/) (2024)
- [Benchmarking LLM Code Generation for Audio Programming with Visual Dataflow Languages](https://mlsanity.com/paper/2409.00856/) (2024)

## Built on by (most cited first, 2 of 2)

- [LLM-based Visual Code Completion for Aerospace Geometric Design](https://mlsanity.com/paper/2606.16806/) (2026)
- [Repository-Level Solidity Code Generation with Large Language Models: From Prompting to Fine-Tuning](https://mlsanity.com/paper/2606.19988/) (2026)

## Abstract (arXiv)

Visual programming languages (VPLs) allow users to create programs through graphical interfaces, which results in easier accessibility and their widespread usage in various domains. To further enhance this accessibility, recent research has focused on generating VPL code from user instructions using large language models (LLMs). Specifically, by employing prompting-based methods, these studies have shown promising results. Nevertheless, such approaches can be less effective for industrial VPLs such as Ladder Diagram (LD). LD is a pivotal language used in industrial automation processes and involves extensive domain-specific configurations, which are difficult to capture in a single prompt. In this work, we demonstrate that training-based methods outperform prompting-based methods for LD generation accuracy, even with smaller backbone models. Building on these findings, we propose a two-stage training strategy to further enhance VPL generation. First, we employ retrieval-augmented fine-tuning to leverage the repetitive use of subroutines commonly seen in industrial VPLs. Second, we apply direct preference optimization (DPO) to further guide the model toward accurate outputs, using systematically generated preference pairs through graph editing operations. Extensive experiments on real-world LD data demonstrate that our approach improves program-level accuracy by over 10% compared to supervised fine-tuning, which highlights its potential to advance industrial automation.