# FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system

arXiv: https://arxiv.org/abs/2410.21349 · HTML: https://mlsanity.com/paper/2410.21349/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 30 of its 54 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Process reward models for reasoning](https://mlsanity.com/alignment/process-reward-models-for-reasoning/)

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [The Faiss library](https://mlsanity.com/paper/2401.08281/) (2024)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [RLTF: Reinforcement Learning from Unit Test Feedback](https://mlsanity.com/paper/2307.04349/) (2023)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback](https://mlsanity.com/paper/2402.01391/) (2024)
- [Compilable Neural Code Generation with Compiler Feedback](https://mlsanity.com/paper/2203.05132/) (2022)
- [WcDT: World-centric Diffusion Transformer for Traffic Scene Generation](https://mlsanity.com/paper/2404.02082/) (2024)
- [CodeFusion: A Pre-trained Diffusion Model for Code Generation](https://mlsanity.com/paper/2310.17680/) (2023)
- [CMAT: A Multi-Agent Collaboration Tuning Framework for Enhancing Small Language Models](https://mlsanity.com/paper/2404.01663/) (2024)
- [Human-centric Reward Optimization for Reinforcement Learning-based Automated Driving using Large Language Models](https://mlsanity.com/paper/2405.04135/) (2024)
- [DDPM-MoCo: Advancing Industrial Surface Defect Generation and Detection with Generative and Contrastive Learning](https://mlsanity.com/paper/2407.03332/) (2024)
- [Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation](https://mlsanity.com/paper/1905.12799/) (2019)
- [SCRNet: a Retinex Structure-based Low-light Enhancement Model Guided by Spatial Consistency](https://mlsanity.com/paper/2305.08053/) (2023)
- [GAgent: An Adaptive Rigid-Soft Gripping Agent with Vision Language Models for Complex Lighting Environments](https://mlsanity.com/paper/2403.10850/) (2024)
- [Neural network based model predictive control of voltage for a polymer electrolyte fuel cell system with constraints](https://mlsanity.com/paper/2406.16871/) (2024)
- [Memory Sequence Length of Data Sampling Impacts the Adaptation of Meta-Reinforcement Learning Agents](https://mlsanity.com/paper/2406.12359/) (2024)

## Built on by (most cited first, 8 of 8)

- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [SCORE: Story Coherence and Retrieval Enhancement for AI Narratives](https://mlsanity.com/paper/2503.23512/) (2025)
- [Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategy](https://mlsanity.com/paper/2501.15167/) (2025)
- [ForgetMe: Evaluating Selective Forgetting in Generative Models](https://mlsanity.com/paper/2504.12574/) (2025)
- [Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures](https://mlsanity.com/paper/2508.10824/) (2025)
- [DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs](https://mlsanity.com/paper/2511.20468/) (2025)
- [Policy Optimized Text-to-Image Pipeline Design](https://mlsanity.com/paper/2505.21478/) (2025)
- [Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generation](https://mlsanity.com/paper/2509.25243/) (2025)

## Abstract (arXiv)

Recently, large language models (LLMs) have achieved significant progress in automated code generation. Despite their strong instruction-following capabilities, these models frequently struggled to align with user intent in coding scenarios. In particular, they were hampered by datasets that lacked diversity and failed to address specialized tasks or edge cases. Furthermore, challenges in supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) led to failures in generating precise, human-intent-aligned code. To tackle these challenges and improve the code generation performance for automated programming systems, we propose Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization (i.e., FALCON). FALCON is structured into two hierarchical levels. From the global level, long-term memory improves code quality by retaining and applying learned knowledge. At the local level, short-term memory allows for the incorporation of immediate feedback from compilers and AI systems. Additionally, we introduce meta-reinforcement learning with feedback rewards to solve the global-local bi-level optimization problem and enhance the model's adaptability across diverse code generation tasks. Extensive experiments demonstrate that our technique achieves state-of-the-art performance, leading other reinforcement learning methods by more than 4.5 percentage points on the MBPP benchmark and 6.1 percentage points on the Humaneval benchmark. The open-sourced code is publicly available at https://github.com/titurte/FALCON.