When AI Reviews Train AI Reviewers: Scientific-Judgment Collapse and Mitigation

LLMs increasingly assist scientific peer review. This creates a feedback loop when generated reviews enter the training data of future LLMs. While recursive training has been studied broadly, its effects on scientific judgments remain poorly understood. We empirically study this effect in a controlled setting by varying synthetic-review exposure while fixing other training conditions. After one recursive step, increasing exposure from 0% to 100% makes recommendation ratings more concentrated, independently generated reviews of the same paper about 11% more similar, and reviews across different papers about 5% less diverse. A further step of recursive training amplifies homogenization when synthetic supervision is reintroduced. We characterize this task-specific phenomenon as scientific-judgment collapse. We also explore a training-free method using contrastive activation steering with a limited reference reviews. Controlled experiments show a partial recovery in recommendation agreement and judgment diversity across different papers, but not among independently generated reviews of the same paper. To explore this approach beyond controlled experiments, we develop TrustReviewer. We train the model on a large-scale curated data and apply activation steering at inference time. On held-out papers, steering increases exact recommendation match from 73.85% to 75.40%, compared with 73.10% for OpenReviewer on the same LLM backbone, while maintaining relatively diverse scientific judgments. Our findings quantify judgment homogenization under recursive training and motivate further research on preserving judgment diversity.