# Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics

arXiv: https://arxiv.org/abs/2603.22018 · HTML: https://mlsanity.com/paper/2603.22018/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 71 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 24 of 24)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [Code Representation Learning At Scale](https://mlsanity.com/paper/2402.01935/) (2024)
- [Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approaches](https://mlsanity.com/paper/2510.04905/) (2025)
- [A large-scale study on research code quality and execution](https://mlsanity.com/paper/2103.12793/) (2021)
- [Deep Just-In-Time Inconsistency Detection Between Comments and Source Code](https://mlsanity.com/paper/2010.01625/) (2020)
- [SciCoQA: Quality Assurance for Scientific Paper--Code Alignment](https://mlsanity.com/paper/2601.12910/) (2026)
- [Code Representation Pre-training with Complements from Program Executions](https://mlsanity.com/paper/2309.09980/) (2023)
- [Large Language Models for Software Engineering: A Reproducibility Crisis](https://mlsanity.com/paper/2512.00651/) (2025)
- [Code Comment Inconsistency Detection with BERT and Longformer](https://mlsanity.com/paper/2207.14444/) (2022)
- [Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modeling](https://mlsanity.com/paper/2511.03404/) (2025)
- [PseudoBridge: Pseudo Code as the Bridge for Better Semantic and Logic Alignment in Code Retrieval](https://mlsanity.com/paper/2509.20881/) (2025)
- [Investigating the Impact of Code Comment Inconsistency on Bug Introducing](https://mlsanity.com/paper/2409.10781/) (2024)

## Built on by (most cited first, 1 of 1)

- [Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detection](https://mlsanity.com/paper/2609.03416/) (2026)

## Abstract (arXiv)

As bioinformatics software is increasingly applied across a broader range of scenarios and the rapid development of large language models (LLMs) further lowers the barriers to software use and development, the composition of the bioinformatics research community is undergoing substantial change. Consequently, a growing number of researchers require a deeper understanding of methodological details and software behavior. In this context, systematically analyzing the relationship between paper descriptions and code implementations is emerging as an important new challenge in the field. To address this challenge, we introduce paper-code consistency analysis as a new research perspective and construct BioCon, the first benchmark dataset for paper-code consistency analysis in bioinformatics. Furthermore, we develop a unified cross-modal analysis framework to systematically investigate this problem from three perspectives: sentence-level detection, cross-modal retrieval, and project-level assessment. Experimental results demonstrate that the proposed framework can effectively model the semantic relationships between scientific publications and software implementations. Further case studies reveal that paper-code inconsistency is not a single phenomenon but arises from multiple underlying causes, among which Author-Perceived Non-Essential Details represents the most prevalent category. These findings suggest that paper-code consistency analysis is not merely a technical problem but also raises broader discussions regarding knowledge dissemination, the boundaries of code disclosure, and community norms. We hope that this work will encourage the bioinformatics community to re-examine the relationship between scientific publications and software implementations while providing a foundation for future research in paper-code consistency analysis.