# Adapting CRISP-DM for Idea Mining: A Data Mining Process for Generating Ideas Using a Textual Dataset

arXiv: https://arxiv.org/abs/2105.00574 · HTML: https://mlsanity.com/paper/2105.00574/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 171 references that are in the map.
- First posted on arXiv 2 May 2021.

## Builds on (most cited first, 13 of 13)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [Accelerating Innovation Through Analogy Mining](https://mlsanity.com/paper/1706.05585/) (2017)
- [The Journal Coverage of Web of Science and Scopus: a Comparative Analysis](https://mlsanity.com/paper/1511.08096/) (2015)
- [A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases](https://mlsanity.com/paper/1305.0377/) (2013)
- [Window-Based Descriptors for Arabic Handwritten Alphabet Recognition: A Comparative Study on a Novel Dataset](https://mlsanity.com/paper/1411.3519/) (2014)
- [A framework to discover potential ideas of new product development from crowdsourcing application](https://mlsanity.com/paper/1502.07015/) (2015)
- [Printed Arabic Text Recognition using Linear and Nonlinear Regression](https://mlsanity.com/paper/1702.01444/) (2017)
- [CASP-DM: Context Aware Standard Process for Data Mining](https://mlsanity.com/paper/1709.09003/) (2017)
- [Refining Student Marks based on Enrolled Modules Assessment Methods using Data Mining Techniques](https://mlsanity.com/paper/2009.06381/) (2020)
- [Mosques Smart Domes System using Machine Learning Algorithms](https://mlsanity.com/paper/2009.10616/) (2020)
- [Intelligent Risk Alarm for Asthma Patients using Artificial Neural Networks](https://mlsanity.com/paper/2009.12175/) (2020)

## Abstract (arXiv)

Data mining project managers can benefit from using standard data mining process models. The benefits of using standard process models for data mining, such as the de facto and the most popular, Cross-Industry-Standard-Process model for Data Mining (CRISP-DM) are reduced cost and time. Also, standard models facilitate knowledge transfer, reuse of best practices, and minimize knowledge requirements. On the other hand, to unlock the potential of ever-growing textual data such as publications, patents, social media data, and documents of various forms, digital innovation is increasingly needed. Furthermore, the introduction of cutting-edge machine learning tools and techniques enable the elicitation of ideas. The processing of unstructured textual data to generate new and useful ideas is referred to as idea mining. Existing literature about idea mining merely overlooks the utilization of standard data mining process models. Therefore, the purpose of this paper is to propose a reusable model to generate ideas, CRISP-DM, for Idea Mining (CRISP-IM). The design and development of the CRISP-IM are done following the design science approach. The CRISP-IM facilitates idea generation, through the use of Dynamic Topic Modeling (DTM), unsupervised machine learning, and subsequent statistical analysis on a dataset of scholarly articles. The adapted CRISP-IM can be used to guide the process of identifying trends using scholarly literature datasets or temporally organized patent or any other textual dataset of any domain to elicit ideas. The ex-post evaluation of the CRISP-IM is left for future study.