A Note on Code Quality Score: LLMs for Maintainable Large Codebases
Not on the map. Why?
Abstract (arXiv)
Maintaining code quality in large-scale software systems presents significant challenges, particularly in settings where a large numbers of engineers work concurrently on a codebase. This paper introduces Code Quality Score (CQS) system to automatically detect issues with a set of code changes and provide actionable insights. At its core, the CQS system is powered by two Llama3 models, fine-tuned (with SFT and offline RL approaches), to a) detect common code quality issues related to coding best practices and b) to provide good ``critiques'' for LLM-generated code review respectively. To maintain good user experience, we layer the system with hand-crafted rules to filter out incorrect responses/hallucinations. Offline evaluations show that our CQS system is able to achieve an impressive precision rate for identifying valid issues. This system has already been rolled out to developers in an industrial scale setting and has consistently achieved 60% week over week user helpfulness rate, demonstrating its effectiveness in a real-world environment. In this paper, we present details of the CQS system along with some learnings on curating developer feedback to create training data for LLM fine-tuning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 1 Aug 2025.
Builds on
20 of 30 references.
References · 20
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2024Self-Rewarding Language Modelscited by 681
- 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
- 2024Critique-out-Loud Reward Modelscited by 106
- 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
- 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
8 more
- 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
- 2022Using Pre-Trained Models to Boost Code Review Automationcited by 63
- 2023LLaMA-Reviewer: Advancing Code Review Automation with Large Language Models through Parameter-Efficient Fine-Tuningcited by 47
- 2022AUGER: Automatically Generating Review Comments with Pre-training Modelscited by 23
- 2023LLMEval: A Preliminary Study on How to Evaluate Large Language Modelscited by 23
- 2025Group Think: Multiple Concurrent Reasoning Agents Collaborating at Token Level Granularitycited by 22
- 2024AI-Assisted Assessment of Coding Practices in Modern Code Reviewcited by 16
- 2022Code Red: The Business Impact of Code Quality -- A Quantitative Study of 39 Proprietary Production Codebasescited by 8
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Rethinking Code Review Workflows with LLM Assistance: An Empirical Study
- 2025Enhancing the Non-Functional Quality Compliance of LLM-Generated Code through Quality-Aware Preference Learning
- 2025Large Language Model Critics for Execution-Free Evaluation of Code Changes
- 2024Software Vulnerability and Functionality Assessment using LLMs
- 2025Are They All Good? Evaluating the Quality of CoTs in LLM-based Code Generation
- 2026Code Health in LLM-Based Test Generation: Effectiveness and Token Efficiency
- 2025Measuring how changes in code readability attributes affect code quality evaluation by Large Language Models
- 2025LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead ↗