Training and Agentic Inference Strategies for LLM-based Manim Animation Generation
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
Generating programmatic animation using libraries such as Manim presents unique challenges for Large Language Models (LLMs), requiring spatial reasoning, temporal sequencing, and familiarity with domain-specific APIs that are underrepresented in general pre-training data. A systematic study of how training and inference strategies interact in this setting is lacking in current research. This study introduces ManimTrainer, a training pipeline that combines Supervised Fine-tuning (SFT) with Reinforcement Learning (RL) based Group Relative Policy Optimisation (GRPO) using a unified reward signal that fuses code and visual assessment signals, and ManimAgent, an inference pipeline featuring Renderer-in-the-loop (RITL) and API documentation-augmented RITL (RITL-DOC) strategies. Using these techniques, this study presents the first unified training and inference study for text-to-code-to-video transformation with Manim. It evaluates 17 open-source sub-30B LLMs across nine combinations of training and inference strategies using ManimBench. Results show that SFT generally improves code quality, while GRPO enhances visual outputs and increases the models' responsiveness to extrinsic signals during self-correction at inference time. The Qwen 3 Coder 30B model with GRPO and RITL-DOC achieved the highest overall performance, with a 94% Render Success Rate (RSR) and 85.7% Visual Similarity (VS) to reference videos, surpassing the baseline GPT-4.1 model by +3 percentage points in VS. Additionally, the analysis shows that the correlation between code and visual metrics strengthens with SFT and GRPO but weakens with inference-time enhancements, highlighting the complementary roles of training and agentic inference strategies in Manim animation generation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 37 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
30 of 37 references.
References · 30
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
18 more
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2023Teaching Large Language Models to Self-Debugcited by 963
- 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
- 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
- 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
- 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
- 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
- 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 89
- 2025Code2Video: A Code-centric Paradigm for Educational Video Generationcited by 23
- 2025TheoremExplainAgent: Towards Video-based Multimodal Explanations for LLM Theorem Understandingcited by 20
- 2025VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learningcited by 18
- 2025ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Codingcited by 12
- 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 12
- 2024LogoMotion: Visually-Grounded Code Synthesis for Creating and Editing Animationcited by 10
- 2025JanusCoder: Towards a Foundational Visual-Programmatic Interface for Code Intelligencecited by 7
- 2025Manimator: Transforming Research Papers into Visual Explanationscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Training and Benchmarking Code Generation for Physics-Inspired Animations
- 2026SFT or RL for Tool-Calling Agents? A Controlled Study Across Data, Method, and Scale
- 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
- 2025Training-Free Group Relative Policy Optimization
- 2025Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation
- 2026Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design
- 2025AdaTooler-V: Adaptive Tool-Use for Images and Videos
- 2024Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement