# Semantic Source Code Segmentation using Small and Large Language Models

arXiv: https://arxiv.org/abs/2507.08992 · HTML: https://mlsanity.com/paper/2507.08992/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Text Segmentation as a Supervised Learning Task](https://mlsanity.com/paper/1803.09337/) (2018)
- [Text Segmentation by Cross Segment Attention](https://mlsanity.com/paper/2004.14535/) (2020)
- [LumberChunker: Long-Form Narrative Document Segmentation](https://mlsanity.com/paper/2406.17526/) (2024)
- [Unsupervised Topic Segmentation of Meetings with BERT Embeddings](https://mlsanity.com/paper/2106.12978/) (2021)
- [Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMs](https://mlsanity.com/paper/2305.12138/) (2023)
- [Topic Segmentation and Labeling in Asynchronous Conversations](https://mlsanity.com/paper/1402.0586/) (2014)
- [Experience with GitHub Copilot for Developer Productivity at Zoominfo](https://mlsanity.com/paper/2501.13282/) (2025)
- [GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding](https://mlsanity.com/paper/2311.09707/) (2023)
- [Unsupervised Dialogue Topic Segmentation in Hyperdimensional Space](https://mlsanity.com/paper/2308.10464/) (2023)
- [CORAL: COde RepresentAtion Learning with Weakly-Supervised Transformers for Analyzing Data Analysis](https://mlsanity.com/paper/2008.12828/) (2020)
- [Logical Segmentation of Source Code](https://mlsanity.com/paper/1907.08615/) (2019)
- [Improving Segmentation for Technical Support Problems](https://mlsanity.com/paper/2005.11055/) (2020)
- [Topic Segmentation of Semi-Structured and Unstructured Conversational Datasets using Language Models](https://mlsanity.com/paper/2310.17120/) (2023)

## Abstract (arXiv)

Source code segmentation, dividing code into functionally coherent segments, is crucial for knowledge retrieval and maintenance in software development. While enabling efficient navigation and comprehension of large codebases, manual and syntactic analysis approaches have become impractical as repositories grow, especially for low-resource languages like R and their research domains (e.g., social sciences, psychology).This paper introduces an automated, domain-specific approach for research R code segmentation using Large and Small Language Models (LLMs/SLMs). It presents two novel approaches and a human-annotated dataset, StatCodeSeg. We explore two distinct approaches: line-by-line analysis with context and range-based segment determination. We experiment with LLMs and fine-tuned SLMs. To support the generalizability of our approaches, we also include experiments on Python code from the computer science domain.Our results show that context-based line-by-line analysis is superior over range-based segmentation.Using smaller language models like CodeBERT and an encoder-only version of CodeT5+ are better than their LLM counterparts. Most notably, these two best-performing models did not see R code during pre-training versus the LLMs but were only fine-tuned on 4,130 lines of manually annotated code.