Professional Documents
Culture Documents
HDPM: An Effective Heart Disease Prediction Model For A Clinical Decision Support System
HDPM: An Effective Heart Disease Prediction Model For A Clinical Decision Support System
ABSTRACT Heart disease, one of the major causes of mortality worldwide, can be mitigated by early
heart disease diagnosis. A clinical decision support system (CDSS) can be used to diagnose the subjects’
heart disease status earlier. This study proposes an effective heart disease prediction model (HDPM) for
a CDSS which consists of Density-Based Spatial Clustering of Applications with Noise (DBSCAN) to
detect and eliminate the outliers, a hybrid Synthetic Minority Over-sampling Technique-Edited Nearest
Neighbor (SMOTE-ENN) to balance the training data distribution and XGBoost to predict heart disease. Two
publicly available datasets (Statlog and Cleveland) were used to build the model and compare the results with
those of other models (naive bayes (NB), logistic regression (LR), multilayer perceptron (MLP), support
vector machine (SVM), decision tree (DT), and random forest (RF)) and of previous study results. The
results revealed that the proposed model outperformed other models and previous study results by achieving
accuracies of 95.90% and 98.40% for Statlog and Cleveland datasets, respectively. In addition, we designed
and developed the prototype of the Heart Disease CDSS (HDCDSS) to help doctors/clinicians diagnose the
patients’/subjects’ heart disease status based on their current condition. Therefore, early treatment could be
conducted to prevent the deaths caused by late heart disease diagnosis.
INDEX TERMS Heart disease, disease prediction model, clinical decision support system, outlier data,
imbalanced data, machine learning.
This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
133034 VOLUME 8, 2020
N. L. Fitriyani et al.: HDPM: Effective HDPM for a CDSS
logistic regression (LR) [16], support vector machine (SVM) and applicability of our proposed model for real-world
[17], and random forest (RF) [18] have been success- case study. It is expected that the developed system
fully used to help as decision making tools for heart dis- can be used as a practical guideline for the healthcare
ease prediction based on individual data. Several studies practitioners.
have also revealed the advantage of a hybrid model which The remainder of this study is organized as follows.
achieved good performance in predicting heart disease such Section II summarized the literature review. Section III
as majority voting of naïve bayes (NB), bayes net (BN), presents the proposed HDPM including datasets description,
RF, and MLP [19], two stacked SVMs [20], and RF with overall design, and modules of the proposed model as well
a linear model [21]. However, in the machine learning as performance evaluation metrics. Section IV discusses the
field, outlier and imbalance data may arise and impact on performance evaluation of proposed model, including the sta-
the performance of the prediction model. Previous studies tistical test and comparison with previous studies. Section V
have reported that by incorporating Density-Based Spatial presents the practical applications of the proposed model in
Clustering of Applications with Noise (DBSCAN)-based the real case scenario. Finally, the concluding remarks and
to detect and eliminate the outlier data [22]–[24], and by future research directions are presented in Section VI.
balancing the distribution of data using a hybrid Synthetic
Minority Over-sampling Technique-Edited Nearest Neighbor II. LITERATURE REVIEW
(SMOTE-ENN) [25]–[28], the prediction models’ perfor- Several studies have reported the development of heart dis-
mances were significantly enhanced. ease diagnosis based on machine learning models with the
To the best of our knowledge, no study has investi- aim of providing an HDPM with enhanced performance.
gated a heart disease prediction model (HDPM) by utilizing Two publicly available heart disease datasets, namely Stat-
DBSCAN, SMOTE-ENN and XGBoost machine learning. log and Cleveland, have been widely used to compare the
Therefore, we propose an effective HDPM for a CDSS which performance of prediction models among researchers. For
consists of DBSCAN-based to detect and eliminate the out- Statlog dataset, a heart disease clinical decision support sys-
liers, SMOTE-ENN to balance the training data distribution tem based on chaos firefly algorithm and rough sets-based
and XGBoost to predict heart disease. Our challenge is to attribute reduction (CFARS-AR) was developed by Long
detect and remove the outlier data and to balance the distri- et al. (2015) [13]. The rough sets were used to reduce the
bution of the training dataset to improve the performance of number of attributes while the chaos firefly algorithm was
the HDPM. Two publicly available datasets (Statlog [29] and used to classify the disease. The developed model was then
Cleveland [30]) were used to build the model and to eval- compared with other models such as NB, SVM and ANN.
uate their performance compared with that of other models The results revealed that the proposed model achieved the
(NB, LR, MLP, SVM, decision tree (DT), and RF) and of highest performance among all the models with accuracy,
previous study results. In addition, we ensured the applica- sensitivity, and specificity of 88.3%, 84.9%, and 93.3%,
bility of the proposed model by designing and implement- respectively. The combination of rough sets-based attributes
ing the model into a Heart Disease CDSS (HDCDSS) to selection and BPNN (RS-BPNN) was proposed by Nahato
diagnose the subjects based on their current condition. The et al. (2015) [14]. With the selected attributes, the pro-
developed HDCDSS is expected to help clinicians diagnose posed RS-BPNN achieved accuracy of up to 90.4%. Dwivedi
the patients effectively and efficiently and thereby improv- (2018) [31] compared six machine learning models (ANN,
ing heart disease clinical decision making. Therefore, early SVM, LR, k-nearest neighbor (kNN), classification tree and
treatment could be conducted to prevent the deaths caused by NB) with various performance metrics. The results showed
late heart disease diagnosis. Contributions of our study can that LR performed better than the other models by achiev-
be summarized as follows. ing up to 85%, 89%, 81%, and 85 for the accuracy, sen-
• Improving accuracy of heart disease prediction model. sitivity, specificity, and precision, respectively. Amin et al.
We proposed HDPM by integrating DBSCAN outlier (2019) [32] performed comparison analysis by identifying
detection, SMOTE-ENN, and XGBoost to improve pre- significant attributes and applying machine learning models
diction accuracy. The HDPM learned from two public (k-NN, DT, NB, LR, SVM, Neural Network (NN) and a
datasets and the trained model was utilized to predict hybrid (voting with NB and LR)). The experiment results
the subjects’ heart disease status based on their current revealed that the hybrid model (voting with NB and LR) with
condition. selected attributes achieved the highest accuracy (87.41%).
• Performance analysis and comparison with state-of- Cleveland heart disease dataset has been widely
the arts models. The proposed HDPM was evaluated used by researchers to generate predictive models.
with other classification models and compared with the Verma et al. (2016) [15] developed a hybrid prediction
results from previous studies. In addition, we presented model based on correlation feature subset (CFS), particle
the statistical evaluation to confirm the significant of our swam optimization (PSO), K-means clustering and MLP.
model as compared to other models. The results showed that the proposed hybrid model achieved
• Real case system development. We designed and devel- accuracy of up to 90.28%. Haq et al. (2018) [16] performed
oped the prototype of the system to show the feasibility a comparative study on a hybrid model based on various
FIGURE 1. The proposed Heart Disease Prediction Model (HDPM) for the Heart Disease Clinical Decision Support System (HDCDSS).
feature selection techniques (relief, minimal-redundancy- None of the aforementioned previous studies have applied
maximal-relevance (mRMR), least absolute shrinkage and outlier detection and data balancing method to improve the
selection operator (LASSO)) and machine learning models accuracy of classification model, especially for the case of
(LR, kNN, ANN, SVM, DT, NB, and RF). Their study heart disease datasets. Thus, in this study we used outlier
revealed that the features reduction affects the performance detection and data balancing methods to improve the model
of the models. The study concluded that a combination of performance. In addition, the XGBoost classifier is then used
Relief-based feature selection and LR-based machine learn- to learn and generate the prediction model.We expect that
ing algorithm (MLA) provides higher accuracy (up to 89%) our proposed model will achieve higher performance than
as compared with other combinations used in the study. that of state-of-the-art models and previous study results.
Saqlain et al. (2019) [17] proposed a technique based on Finally, we also design and develop the HDCDSS to help doc-
mean Fisher score feature selection algorithm (MFSFSA) tors/clinicians diagnose the patients’/subjects’ heart disease
and SVM classification model. The selected features are status based on their current condition. Thus, early treatment
based on the higher Fisher score than the mean score. Then, could be conducted to prevent the risks further.
SVM used the selected feature subset to learn and calculate
the MCC through a validation process. The study revealed III. MATERIALS AND METHODS
that the combination of FSFSA and SVM generates accu- The proposed HDPM was developed to provide high perfor-
racy, sensitivity, and specificity of up to 81.19%, 72.92%, mance prediction in the presence or absence of heart disease
and 88.68%, respectively. Latha and Jeeva (2019) [19] pro- given the current condition of the subjects. The flow-chart
posed a hybrid model with majority voting of NB, BN, RF, in Figure 1 shows how the proposed HDPM is developed.
and MLP. The proposed model achieved an accuracy of up First, the heart disease datasets are collected. Second, the data
to 85.48%. pre-processing for data transformation and feature selection
Ali et al. (2019) [20] proposed two stacked SVMs to are conducted. Third, the DBSCAN-based outlier detection
improve the diagnosis process. The first SVM was used to method is applied to find the outlier data given the optimal
remove the non-relevant features and the second to predict parameter. Fourth, the detected outlier data are then removed
heart disease. The results revealed that the proposed model from the training dataset. Fifth, the data balancing based on
achieved better performance than other models and previous SMOTE-ENN method is used to balance the training dataset.
study results. Mohan et al. (2019) [21] introduced a hybrid RF Sixth, the XGBoost-based MLA is used to learn from the
with a linear model (HRFLM) to enhance the performance of training dataset and generate the HDPM. Finally, the per-
the HDPM. They found that the proposed method achieved formance metrics are presented to evaluate the performance
accuracy, precision, sensitivity, f-measure and specificity of of the proposed model and the generated HDPM is then
up to 88.4%, 90.1%, 92.8%, 90%, and 82.6%, respectively. implemented within the CDSS. In our study, we utilized
Recently, Gupta et al. (2020) [18] developed a machine 10-fold cross-validation method to avoid the overfitting.
intelligence framework consisting of factor analysis of mixed Cross-validation allows the models to learn from different
data (FAMD) and RF-based MLA. The FAMD was used to sets of training data by repeated sampling; hence maximizing
find the relevant features and the RF to predict the disease. the data used for validation and possibly, helping to pre-
The experimental results showed that the proposed method vent from overfitting. Previous study has demonstrated that
outperformed other models and previous study results by 10-fold cross-validation can be used to maintain the bias-
achieving the accuracy, sensitivity, and specificity of up to variance trade-off which eventually provide the generalized
93.44%, 89.28%, and 96.96%, respectively. model and protect against overfitting [33], [34].
TABLE 1. The detailed dataset attributes description and distribution (mean and standard deviation (STD)) for dataset I (Statlog).
The detailed steps, including datasets and modules 79 raw attributes, although only 13 attributes are used, and
descriptions, and the performances metrics are presented in one attribute as an output class. We removed 6 subjects’ data
the following subsections. In addition, the performance of the due to missing values and used the remaining 297 data in the
proposed model with the state-of-the-art models is evaluated pre-processing stage. The original class value is a multi-class
and the results are presented in the results and discussion variable with the value range from 0 to 4. The 0 value is
section. Finally, we ensure the applicability of the proposed used to represent the absence of heart disease while the
model by embedding the HDPM into the HDCDSS to diag- values from 1 to 4 are used to represent the presence of heart
nose the subjects’ heart disease status based on their current disease with its stage condition. In this study, we followed
condition. previous studies [16]–[21, [32] in converting the class value
from a multi-class variable to a binary-class variable. The
A. HEART DISEASE DATASET final class variable is set to 0 if heart disease is not present
We used two heart disease datasets (Statlog and Cleveland; in the subject and to 1 for all the subjects who have been
termed datasets I and II, respectively) to investigate how heart diagnosed as having heart disease. We pre-processed the data
disease can be identified by applying the machine learning by applying the previous rule to the records. Finally, after data
model. The proposed model is then applied to those two pre-processing, the final dataset II consists of 297 subjects
datasets and with the expectation of providing a general and with 137 and 160 subjects being labelled with the presence
robust HDPM. (positive class) and absence (negative class) of heart disease,
The University of California Irvine (UCI) Repository Stat- respectively. A detailed attributes description (including data
log Heart Disease database website presents dataset I to type and range) and distribution (mean and STD) for dataset II
investigate heart disease [29]. The original dataset consists is given in Table 2.
of 270 subjects, 13 attributes and one output class (120 and For both datasets, the absence and presence of heart disease
150 subjects are labelled with the presence (positive class) are treated as negative (0) and positive (1), respectively. The
and absence (negative class) of heart disease, respectively). correlation between attributes can affect the performance of
There are no missing values in dataset I. A detailed attributes the machine learning model. Data correlation by utilizing
description (including data type and range) and distribution Pearson’s Correlation Coefficient (PCC) can be used as a cal-
(mean and standard deviation (STD)) for dataset I are given culation tool to determine the relationship between attributes.
in Table 1. PCC varies from −1 to +1, with a positive and a nega-
Dr. Robert Detrano, M.D., provided dataset II (Cleve- tive value indicating a highly positive and highly negative
land Heart Disease dataset) to investigate heart disease correlation between the variables, respectively, and a value
that was collected from the V.A. Medical Center, Long close to zero indicating a low correlation between them. The
Beach and Cleveland Clinic Foundation in California, United heatmap correlation between attributes for datasets I and II
States [30]. The original dataset comprises 303 subjects and are given in Figure 2(a) and 2(b), respectively. The gray color
TABLE 2. The detailed dataset attributes description and distribution (mean and standard deviation (STD)) for dataset II (Cleveland).
FIGURE 2. Heatmap of attributes correlation for (a) dataset I (Statlog) and (b) dataset II (Cleveland).
indicates that the correlation is close to 0, while the red and select the most important attribute to improve the model per-
blue colors indicate that the correlation between variables is formance for the two datasets [37], [38]. Figure 3(a) and 3(b)
close to +1 and −1, respectively. The attributes chol and fbs show the attribute significant score based on the IG
are seen to have a correlation that is close to 0 toward the method for datasets I and II, respectively. In this case,
attribute class, which suggests that both only have a small or both datasets have the same lowest attributes scores (chol,
even no correlation with the attribute class. Thus, we could trestbps, and fbs), which we therefore removed from both
possibly remove these features to improve the performance datasets, and used the remaining attributes (age, sex, cp,
of our proposed model. restecg, thalach, exang, oldpeak, slope, ca, and thal) for
In addition, we applied attribute selection by using the further analysis. We expect that by using the two datasets
Information Gain (IG) method [35] in Weka V3.8 [36] to and the selected attributes, our proposed model will be
FIGURE 3. Attribute significance score provided by the Information Gain (IG) method for (a) dataset I (Statlog) and (b) dataset II (Cleveland).
FIGURE 4. An illustration of (a) eps, core, border and outlier point and (b) DBSCAN cluster model with MinPts = 5.
sufficiently robust for predicting heart disease with high The ‘‘border point’’ y is defined as the number of neigh-
performance. boring data points is less than MinPts, but y belongs to the
neighboring core data point of x. Finally, the ‘‘outlier point’’
B. DBSCAN-BASED OUTLIER DATA DETECTION AND z is marked as a point z is neither a core point nor a border
REMOVAL point. Figure 4(a) illustrates eps, core x, border y, and outlier
In this study, we utilized DBSCAN [39] to cluster and detect z point using MinPts = 5. As can be seen in Figure 4(b),
the outliers from both training datasets. The goal of DBSCAN the point B and C are border point, A is a core point, and
is to find the dense regions which can be identified by the N is a noise point. Arrows indicate direct density reachability.
number of objects that are close to a specific point (core Point B and C are density connected, because both are density
point) and the points that are outside the regions are treated as reachable from point A. N is not density reachable and do not
outliers. In general, two parameters need to be determined for belong to any cluster (with MinPts = 5), and thus considered
DBSCAN: epsilon (eps) and minimum points (MinPts). The to be a noise point or outlier. First, the algorithm checks
eps is defined as the neighborhood radius around a point of x the specific point (any point) to be considered as a core
(ε-neighborhood) while the MinPts is defined as the mini- point or not. The core point is if at least MinPts points are
mum number of neighboring data points within the eps. There within the eps of it. The border points are the points that
are three points that can be used to determine the normal and can be reached from core point (within distance eps from
outlier data are core point, border point, and outlier point. core point). Next, the core and border points are becoming
A ‘‘core point’’ x is marked as any point that has a number of cluster and marked as visited points by the algorithm. Finally,
neighboring data points either greater than or equal to MinPts. the algorithm keeps iterating to check other unvisited point
FIGURE 5. Optimal eps value using 5-NN and DBSCAN outlier detection result for datasets I (Statlog) (a), (b) and II (Cleveland) (c), (d), respectively.
(to be considered as core point) to find the unvisited border respectively. We found that the ‘‘knee’’ appears at around
points. The points that are not belonging to the clusters are the distance of 9 and 8 for datasets I and II, respectively.
considered as outlier. The detailed pseudocode for DBSCAN Furthermore, we applied the DBSCAN method by using
is presented in Algorithm 1. MinPts = 5, eps = 9 and MinPts = 5, eps = 8 for datasets I
The optimal eps value is calculated by averaging the dis- and II, respectively. Figure 5(b) and (d) show the results of
tance of every point to its kNN. The value of k corresponds to DBSCAN implementation for datasets I and II visualized
the MinPts value, which is defined by the user. In this study, in two-dimensional graphs. The results showed that in both
we followed previous studies [40]–[43] to utilize 5-nearest datasets, the DBSCAN clustered the data into a single cluster
neighbors (5-NN) to find the optimal eps value. Most of the as cluster 1 and the un-clustered data (with x symbol) are
previous studies utilized MinPts = 5 and optimized their treated as outliers (see Figure 5(b) and (d)). The optimal
eps value based on MinPts. Finally, according to Ester et al. parameters and the final outlier data for both datasets are pre-
(1996) [39], the eps can be obtained by presenting k-dist sented in Table 3. Finally, we removed all the detected outlier
graph. First, k-distances are visualized as a k-dist graph and data in each training dataset and used the remaining normal
shown in ascending order to find the ‘‘knee’’ value where a data for further analysis. In addition, we performed experi-
sharp change appears beside the k-distance curve for the opti- mental analysis to find the impact of outlier removal on the
mal eps value estimation. We implemented the calculation performance of the model. Figure 6 shows the impact of out-
of kNNs and DBSCAN in R programming V3.5.1 and used lier data elimination based on DBSCAN as compared to orig-
R packages such as fpc V2.2-2 and DBSCAN V1.1-3. inal data. Outlier removal based on DBSCAN significantly
Figure 5(a) and (c) show the sorted 5-NN distribu- improved the model accuracy for all datasets, from accuracy
tion graph and optimal eps value for datasets I and II, 80.74%, 80.03% to 85.41%, 85.26% for dataset I, and II,
Algorithm 1 DBSCAN Pseudocode TABLE 3. The parameters and result of DBSCAN-based outlier detection.
Input: dataset, D; minimum point, minPts; radius, eps
Output: clustered C and un-clustered data UC
for each sample point SP in dataset D do
if SP is not visited then
mark SP as visited
neigbrPts ← samples points in ε-neighborhood of SP TABLE 4. SMOTE-ENN data balancing results.
FIGURE 8. Data distribution of attributes ‘‘age’’ and ‘‘thalach’’ before and after SMOTE-ENN implementation for each dataset I (Statlog) (a), (b) and
dataset II (Cleveland) (c), (d), respectively.
the final performance metric being the average. We imple- and XGBoost. In addition, the following five performance
mented all the classification models in Python V3.6.5 by metrics are measured. Accuracy (acc) is calculated as
utilizing three libraries: sklearn V0.20.2, imbalanced-learn
TP + TN
V0.4.3 and XGBoost V0.81. We performed the experiments acc = , (2)
on a computer with Intel Core i7-4790 (3.60 GHz × 8 cores), TP + FN + FP + TN
16 GB RAM that runs with Windows 10 Pro 64-bit. The precision (pre) is calculated as
sklearn library is an open source python programming tool
for machine learning, the imbalanced-learn library is also an TP
pre = , (3)
open source python tool-box that consists of several methods TP + FP
to deal with imbalanced data, and the XGBoost library is an
open source tool that implements the XGBoost algorithms in recall/sensitivity/true positive rate (rec/sen/TPR) is calculated
several programming languages, including Python. To sim- as
plify the implementation of the experimentations, we used TP
default parameters provided by sklearn, imbalanced-learn rec = , (4)
TP + FN
FIGURE 9. ROC curve visualization to compare the proposed model with other models for datasets (a) I (Statlog) and (b) II (Cleveland).
model. The best model is achieved when the value of AUC is t (tabulated) = 2.78 and collected the h, p-value, and t
close or equal to 1. Figure 9 shows that the proposed HDPM (calculated) values for all datasets. The null hypothesis is
achieved higher AUC score than that of other models of up accepted when the paired t-test return value of h = 0, and
to 1.00 and 1.00 for datasets I and II, respectively, which the null hypothesis is rejected if h = 1, which indicates
confirmed that the proposed model outperformed other state- a significance different between the proposed HDPM and
of-the-art models. the existing one. This could be supported by evidence that
In addition, we followed a previous study [49] to eval- the p-value is less than the significance level (0.05) and t
uate the performance of the model using statistical-based (calculated) is greater than t (tabulated). In Table 7, show-
significance testing to prove the significance of our proposed ing the paired t-test result for both datasets, the proposed
HDPM as compared with other state-of-the-art models. The HDPM is significantly different from the other models since
paired t-test [50], [51] was applied to statistically test the for all datasets, h = 1, p-value < significance level, and
significance between the proposed HDPM and other state- t (calculated) > t (tabulated). Therefore, the proposed model
of-the-art models. We defined h = 0, i.e., the null hypothe- has significant different as compared with other state-of-the-
sis, as being no significance different between the proposed art models.
HDPM and other existing models. We performed 10-fold
cross validation to collect ten accuracy data for all the models B. BENCHMARK WITH PREVIOUS STUDY RESULTS
in Python V3.6.5 and applied the paired t-test using Scipy In this section, we performed comparison study of our pro-
V1.2.0 library. We defined the significance level = 0.05, posed HDPM with the results from previous studies. It should
TABLE 7. The results of paired t-test for datasets I (Statlog) and II (Cleveland).
be noted that since we utilized the same datasets, we directly In terms of accuracy, the proposed HDPM achieved the
took the results from previous studies without implementing highest accuracy with an average improvement of 8.12%
their techniques. The detailed comparison results with previ- as compared with previous study results. Overall, we can
ous studies for datasets I and II are given in Table 8 and 9, conclude that our proposed method outperformed all the
respectively. previous study results in terms of accuracy, f-measure, MCC
Previous studies have utilized the Statlog dataset for gen- and AUC.
erating the machine learning model to diagnose the heart In addition, several researchers have also used the Cleve-
disease. Long et al. (2015) [13] proposed the CFARS-AR land dataset to predict heart disease. Verma et al. (2016) [15]
and achieved acc = 88.3% and rec/sen = 84.9%. Nahato developed a hybrid model with CFS selection, PSO, K-means
et al. (2015) [14] used the rough set method with RS-BPNN clustering and MLP and achieved acc = 90.28%. Haq et al.
and achieved acc = 90.40%, rec/sen = 94.67% and AUC = (2018) [16] proposed a hybrid system using Relief-based fea-
0.92. Dwivedi (2018) [31] used LR and achieved acc = ture selection and LR, and achieved acc = 89%, rec/sec =
85%, pre = 85%, rec/sec = 89%, and f = 87%. Amin 77%, MCC = 0.89, and AUC = 0.88. Saqlain et al.
et al. (2019) [32] utilized the voting method with NB and (2019) [17] used MFSFSA and SVM and achieved acc =
LR and achieved acc = 87.41%. The proposed HDPM 81.19%, rec/sec = 72.92%, and MCC = 0.85. Latha and
achieved acc = 95.90%, pre = 97.14%, rec/sec = Jeeva (2019) [19] used majority voting with NB, BN, RF,
94.67%, f = 95.35%, MCC = 0.92, and AUC = 1.00. and MLP and achieved acc = 85.48%. Ali et al. (2019) [20]
FIGURE 10. Heart Disease Clinical Decision Support System (HDCDSS) (a) architecture framework, (b) diagnosis form, and (c) diagnosis result.
used stacked SVMs and achieved acc = 92.22%, rec/sec = data pre-processing and training/testing approaches. In addi-
82.92%, and MCC = 0.85. Mohan et al. (2019) [21] devel- tion, the prediction model performance depends on several
oped HRFLM and achieved acc = 88.4%, pre = 90.1%, factors such as features selections, data types and its size,
rec/sec = 92.8%, and f = 90%. Gupta et al. (2020) [18] noise filtering, hyperparameters, data sampling, model selec-
utilized the FAMD-based feature extraction and RF algo- tion, etc. Therefore, these general comparison (as presented
rithm and achieved acc = 93.44%, rec/sec = 89.28%, in Table 8 and 9) cannot be used as the main evidence to
f = 92.59%, MCC = 0.87, and AUC = 0.93. Finally, conclude the performance of given prediction models but it
the proposed HDPM achieved acc = 98.40%, pre = 98.57%, can be used simply as a general comparison between the
rec/sec = 98.33%, f = 98.32%, MCC = 0.97, and proposed HDPM and previous studies.
AUC = 1.00. In terms of accuracy, the proposed HDPM
achieved the highest accuracy with an average improvement
of 9.83% as compared with previous study results. Overall, V. APPLICATION FOR THE HEART DISEASE CLINICAL
we can conclude that our proposed method outperformed all DECISION SUPPORT SYSTEM (HDCDSS)
the previous study results in all six-performance metrics (acc, The prototype of the web-based Heart Disease Clinical Deci-
pre, rec/sen, f , MCC, and AUC). sion Support System (HDCDSS) was developed to pro-
It should be noted that a direct comparison of the presented vide a simple and convenient way for medical clinicians to
results is not fair since they have been derived by different diagnose subjects/patients based on their current condition.
VOLUME 8, 2020 133047
N. L. Fitriyani et al.: HDPM: Effective HDPM for a CDSS
The HDCDSS was developed in Python V3.6.5 by utilizing the unbalanced training dataset and XGBoost MLA was
Flask V1.0.2 as a Python Web Server Gateway Interface adopted to learn and generate the prediction model. Two pub-
(WSGI) with Bootstrap V3.3.7 for data representation, while licly available datasets of heart disease were utilized by pro-
the proposed HDPM was loaded using Joblib V0.14.1 and duce the generalized prediction model. We performed evalu-
XGBoost V0.81. The patients’ data and the prediction results ation analysis of our proposed model with other classification
were stored into MongoDB by using Pymongo V3.7.1. Mon- models and the results from previous studies. In addition,
goDB was selected since it has been widely adopted in the we presented the statistical evaluation to confirm the signif-
healthcare field [52], [53]. As illustrated in Figure 10(a), clin- icant of our model as compared to other models. The exper-
icians can access the HDCDSS through their web-browser imental results confirmed that the proposed model achieved
in the same local network since the medical data are confi- better performance than that of state-of-the-art models and
dential information and cannot be stored in the cloud. The previous study results, by achieving an accuracy up to 95.90%
personal data such as patient id (id), age, and gender are then and 98.40% for datasets I and II, respectively. In addition,
combined with the diagnosis data, such as resting electro- the statistical-based analysis result also showed the signifi-
cardiographic result (restecg), maximum heart rate (thalach), cant improvement for the proposed model as compared with
exercise induced angine (exang), ST depression induced by the other models.
exercise relative to rest (oldpeak), slope of the peak exercise Furthermore, we also designed and developed the proposed
ST segment (slope), number of major vessels (0-3) colored by HDPM into the Heart Disease Clinical Decision Support
fluoroscopy (ca), and defect type (thal), and then transmitted System (HDCDSS) to diagnose the subjects’/patients’ heart
into a secure web server through an application programming disease status effectively and efficiently. The HDCDSS gath-
interface (API) and stored in a database. The proposed HDPM ered the patient data combined with other diagnosis data and
generated from datasets I (Statlog) and II (Cleveland) is then transmitted them to a secure web server. All the transmitted
used to predict the subjects’ heart disease status based on the diagnosis data were then stored into MongoDB, which can
inputted data, and the prediction result is then sent back to the effectively provide timely response with rapidly increasing
HDCDSS’s diagnosis result interface. medical data. The proposed HDPM was then loaded to diag-
Figure 10(b) shows the HDCDSS diagnosis form in which nose the patients’ current heart disease status, which was
clinicians can fill out the patients’ information, including later sent back to the HDCDSS’s diagnosis result interface.
their currents conditions. Once all the input fields are filled, Thus, the developed HDCDSS is expected to help clinicians
the user can press the ‘‘diagnose’’ button to send all the to diagnose patients and improving heart disease clinical deci-
data to the secure web server, which loads the trained pro- sion making effectively and efficiently. Finally, the overall
posed HDPM to diagnose the subjects’ heart disease status. designed and developed HDCDSS in this study can be used
Figure 10(c) shows the diagnosis result interface after sending as a practical guideline for the healthcare practitioners.
the data to the web server. The result includes the previ- In the future, we will consider the comparison of other data
ously submitted data and the status (presence or absence) sampling with the model hyper-parameters and broader med-
of heart disease. The developed HDCDSS is expected to ical datasets. In addition, a comparison and analysis study
help clinicians to diagnose patients and improving heart with different outlier detection methods could be further
disease clinical decision making effectively and efficiently. investigated. Furthermore, with the increasing concerns about
Therefore, early treatment could be conducted to prevent the privacy, security and time-sensitive applications, edge com-
deaths caused by late heart disease diagnosis. This proto- puting and edge device concepts could be further studied with
type/demonstration is only limited to the specific datasets; the goal of improving the medical clinical decision support
therefore, the trained prediction model cannot be applied for system. In this study, we have not obtained any feedback from
other demographic patients/subjects. Once we have collected heart specialist yet. In the future, once specific demographic
more complex datasets, it could improve the predictive per- dataset (from Korea) is collected, the comments from local
formance for wider demographic patients/subjects. In addi- heart specialist for verifying dataset and prediction model
tion, we have not applied the developed model in the clinical could be presented.
trial due to limitation of the dataset. In our case, we have used
the dataset based on specific demographic patient (USA). The ACKNOWLEDGMENT
clinical trial could be applied to our model once we gather This article is a tribute made of deep respect of a wonder-
another demographic patient (for example in Korea) and it is ful person, friend, advisor, and supervisor, Yong-Han Lee
beyond the scope of our current study. (1965–2017).
[3] Statistics Korea. (2018). Causes of Death Statistics in 2018. [19] C. B. C. Latha and S. C. Jeeva, ‘‘Improving the accuracy of pre-
[Online]. Available: http://kostat.go.kr/portal/eng/pressReleases/8/ diction of heart disease risk based on ensemble classification tech-
10/index.board?bmode=read&bSeq=&aSeq=378787 niques,’’ Inform. Med. Unlocked, vol. 16, Jan. 2019, Art. no. 100203, doi:
[4] World Health Organization. (2017). Cardiovascular Diseases 10.1016/j.imu.2019.100203.
(CVDs). [Online]. Available: https://www.who.int/news-room/fact- [20] L. Ali, A. Niamat, J. A. Khan, N. A. Golilarz, X. Xingzhong, A. Noor,
sheets/detail/cardiovascular-diseases-(cvds) R. Nour, and S. A. C. Bukhari, ‘‘An optimized stacked support vec-
[5] P. Greenland, J. S. Alpert, G. A. Beller, E. J. Benjamin, M. J. Budoff, tor machines based expert system for the effective prediction of heart
Z. A. Fayad, E. Foster, M. A. Hlatky, J. M. Hodgson, F. G. Kushner, failure,’’ IEEE Access, vol. 7, pp. 54007–54014, 2019, doi: 10.1109/
M. S. Lauer, L. J. Shaw, S. C. Smith, A. J. Taylor, W. S. Weintraub, and ACCESS.2019.2909969.
N. K. Wenger, ‘‘2010 ACCF/AHA guideline for assessment of cardiovas- [21] S. Mohan, C. Thirumalai, and G. Srivastava, ‘‘Effective heart disease
cular risk in asymptomatic adults: A report of the American college of prediction using hybrid machine learning techniques,’’ IEEE Access, vol. 7,
cardiology foundation/American heart association task force on practice pp. 81542–81554, 2019, doi: 10.1109/ACCESS.2019.2923707.
guidelines,’’ Circulation, vol. 122, no. 25, pp. e584–e636, Dec. 2010, doi: [22] X. Liu, Q. Yang, and L. He, ‘‘A novel DBSCAN with entropy and prob-
10.1161/CIR.0b013e3182051b4c. ability for mixed data,’’ Cluster Comput., vol. 20, no. 2, pp. 1313–1323,
Jun. 2017, doi: 10.1007/s10586-017-0818-3.
[6] J. Perk et al., ‘‘European guidelines on cardiovascular disease prevention in [23] C.-H. Lin, K.-C. Hsu, K. R. Johnson, M. Luby, and Y. C. Fann, ‘‘Applying
clinical practice (version 2012): The fifth joint task force of the European density-based outlier identifications using multiple datasets for validation
society of cardiology and other societies on cardiovascular disease pre- of stroke clinical outcomes,’’ Int. J. Med. Inform., vol. 132, Dec. 2019,
vention in clinical practice (constituted by representatives of nine soci- Art. no. 103988, doi: 10.1016/j.ijmedinf.2019.103988.
eties and by invited experts) ∗ developed with the special contribution [24] Z. H. Ismail, A. K. K. Chun, and M. I. S. Razak, ‘‘Efficient herd—
of the European association for cardiovascular prevention & rehabilitation Outlier detection in livestock monitoring system based on density—Based
(EACPR),’’ Eur. Heart J., vol. 33, no. 13, pp. 1635–1701, Jul. 2012, doi: spatial clustering,’’ IEEE Access, vol. 7, pp. 175062–175070, 2019, doi:
10.1093/eurheartj/ehs092. 10.1109/ACCESS.2019.2952912.
[7] G.-M. Park and Y.-H. Kim, ‘‘Model for predicting cardiovascular disease: [25] G. E. A. P. A. Batista, R. C. Prati, and M. C. Monard, ‘‘A study of the
Insights from a Korean cardiovascular risk model,’’ Pulse, vol. 3, no. 2, behavior of several methods for balancing machine learning training data,’’
pp. 153–157, 2015, doi: 10.1159/000438683. ACM SIGKDD Explor. Newslett., vol. 6, no. 1, pp. 20–29, Jun. 2004, doi:
[8] G. J. Njie, K. K. Proia, A. B. Thota, R. K. C. Finnie, D. P. Hopkins, 10.1145/1007730.1007735.
S. M. Banks, D. B. Callahan, N. P. Pronk, K. J. Rask, D. T. Lackland, [26] T. Le, M. Lee, J. Park, and S. Baik, ‘‘Oversampling techniques for
and T. E. Kottke, ‘‘Clinical decision support systems and prevention,’’ bankruptcy prediction: Novel features from a transaction dataset,’’ Sym-
Amer. J. Preventive Med., vol. 49, no. 5, pp. 784–795, Nov. 2015, doi: metry, vol. 10, no. 4, p. 79, Mar. 2018, doi: 10.3390/sym10040079.
10.1016/j.amepre.2015.04.006. [27] T. Le and S. Baik, ‘‘A robust framework for self-care problem identification
[9] V. Sintchenko, E. Coiera, J. R. Iredell, and G. L. Gilbert, ‘‘Comparative for children with disability,’’ Symmetry, vol. 11, no. 1, p. 89, Jan. 2019, doi:
impact of guidelines, clinical data, and decision support on prescrib- 10.3390/sym11010089.
ing decisions: An interactive Web experiment with simulated cases,’’ [28] T. Le, M. T. Vo, B. Vo, M. Y. Lee, and S. W. Baik, ‘‘A hybrid
J. Amer. Med. Inform. Assoc., vol. 11, no. 1, pp. 71–77, Jan. 2004, doi: approach using oversampling technique and cost-sensitive learning for
10.1197/jamia.M1166. bankruptcy prediction,’’ Complexity, vol. 2019, pp. 1–12, Aug. 2019, doi:
[10] K. Kawamoto, C. A. Houlihan, E. A. Balas, and D. F. Lobach, ‘‘Improving 10.1155/2019/8460934.
clinical practice using clinical decision support systems: A systematic [29] Statlog (Heart) Data Set. Accessed: Oct. 2, 2019. [Online]. Available:
review of trials to identify features critical to success,’’ BMJ, vol. 330, http://archive.ics.uci.edu/ml/datasets/statlog+(heart)
[30] Heart Disease Data Set. Accessed: Oct. 2, 2019. [Online]. Available:
no. 7494, p. 765, Apr. 2005, doi: 10.1136/bmj.38398.500764.8F.
https://archive.ics.uci.edu/ml/datasets/Heart+Disease
[11] H. B. Bosworth, M. K. Olsen, T. Dudley, M. Orr, M. K. Goldstein, [31] A. K. Dwivedi, ‘‘Performance evaluation of different machine learning
S. K. Datta, F. McCant, P. Gentry, D. L. Simel, and E. Z. Oddone, ‘‘Patient techniques for prediction of heart disease,’’ Neural Comput. Appl., vol. 29,
education and provider decision support to control blood pressure in pri- no. 10, pp. 685–693, May 2018, doi: 10.1007/s00521-016-2604-1.
mary care: A cluster randomized trial,’’ Amer. Heart J., vol. 157, no. 3, [32] M. S. Amin, Y. K. Chiam, and K. D. Varathan, ‘‘Identification of significant
pp. 450–456, Mar. 2009, doi: 10.1016/j.ahj.2008.11.003. features and data mining techniques in predicting heart disease,’’ Telemat-
[12] J. Hunt, J. Siemienczuk, W. Gillanders, B. LeBlanc, Y. Rozenfeld, ics Inform., vol. 36, pp. 82–93, Mar. 2019, doi: 10.1016/j.tele.2018.11.007.
K. Bonin, and G. Pape, ‘‘The impact of a physician-directed health infor- [33] R. Kohavi, ‘‘A study of cross-validation and bootstrap for accuracy
mation technology system on diabetes outcomes in primary care: A pre- estimation and model selection,’’ in Proc. 14th Int. Joint Conf. Artif.
and post-implementation study,’’ J. Innov. Health Inform., vol. 17, no. 3, Intell. (IJCAI), Montreal, QC, Canada, vol. 2, Aug. 1995, pp. 1137–1145.
pp. 165–174, Sep. 2009, doi: 10.14236/jhi.v17i3.731. [Online]. Available: http://ijcai.org/Proceedings/95-2/Papers/016.pdf
[13] N. C. Long, P. Meesad, and H. Unger, ‘‘A highly accurate firefly based [34] R. Kohavi and D. Wolpert, ‘‘Bias plus variance decomposition for zero-one
algorithm for heart disease prediction,’’ Expert Syst. Appl., vol. 42, no. 21, loss functions,’’ in Proc. 13th Int. Conf. Mach. Learn., San Francisco, CA,
pp. 8221–8231, Nov. 2015, doi: 10.1016/j.eswa.2015.06.024. USA, 1996, pp. 275–283.
[14] K. B. Nahato, K. N. Harichandran, and K. Arputharaj, ‘‘Knowledge mining [35] J. Han and M. Kamber, Data Mining: Concepts and Techniques. San Diego,
from clinical datasets using rough sets and backpropagation neural net- CA, USA: Elsevier, 2012.
work,’’ Comput. Math. Methods Med., vol. 2015, pp. 1–13, Mar. 2015, doi: [36] Weka 3: Data Mining Software in Java. Accessed: Oct. 19, 2019. [Online].
10.1155/2015/460189. Available: https://www.cs.waikato.ac.nz/ml/weka/
[37] A. L. Blum and P. Langley, ‘‘Selection of relevant features and examples
[15] L. Verma, S. Srivastava, and P. C. Negi, ‘‘A hybrid data mining model
in machine learning,’’ Artif. Intell., vol. 97, pp. 245–271, Dec. 1997, doi:
to predict coronary artery disease cases using non-invasive clinical data,’’
10.1016/S0004-3702(97)00063-5.
J. Med. Syst., vol. 40, no. 7, p. 178, Jul. 2016, doi: 10.1007/s10916-016-
[38] I. Guyon and A. Elisseeff, ‘‘An introduction to variable and feature selec-
0536-z.
tion,’’ J. Mach. Learn. Res., vol. 3, pp. 1157–1182, Mar. 2003.
[16] A. U. Haq, J. P. Li, M. H. Memon, S. Nazir, and R. Sun, ‘‘A hybrid intel- [39] M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, ‘‘A density-based algorithm
ligent system framework for the prediction of heart disease using machine for discovering clusters in large spatial databases with noise,’’ in Proc.
learning algorithms,’’ Mobile Inf. Syst., vol. 2018, pp. 1–21, Dec. 2018, 2nd Int. Conf. Knowl. Discovery Data Mining, Portland, OR, USA, 1996,
doi: 10.1155/2018/3860146. pp. 226–231.
[17] S. M. Saqlain, M. Sher, F. A. Shah, I. Khan, M. U. Ashraf, M. Awais, [40] M. Ijaz, G. Alfian, M. Syafrudin, and J. Rhee, ‘‘Hybrid prediction
and A. Ghani, ‘‘Fisher score and matthews correlation coefficient-based model for type 2 diabetes and hypertension using DBSCAN-based out-
feature subset selection for heart disease diagnosis using support vector lier detection, synthetic minority over sampling technique (SMOTE),
machines,’’ Knowl. Inf. Syst., vol. 58, no. 1, pp. 139–167, Jan. 2019, doi: and random forest,’’ Appl. Sci., vol. 8, no. 8, p. 1325, Aug. 2018, doi:
10.1007/s10115-018-1185-y. 10.3390/app8081325.
[18] A. Gupta, R. Kumar, H. S. Arora, and B. Raman, ‘‘MIFH: A [41] G. Alfian, M. Syafrudin, and J. Rhee, ‘‘Real-time monitoring system
machine intelligence framework for heart disease diagnosis,’’ IEEE using smartphone-based sensors and NoSQL database for perishable
Access, vol. 8, pp. 14659–14674, 2020, doi: 10.1109/ACCESS.2019. supply chain,’’ Sustainability, vol. 9, no. 11, p. 2073, Nov. 2017, doi:
2962755. 10.3390/su9112073.
[42] M. Syafrudin, G. Alfian, N. Fitriyani, and J. Rhee, ‘‘Performance analysis industrial applications. He was selected and invited to participate with the
of IoT-based sensor, big data processing, and machine learning model World Class Scholar Symposium (SCKD) Event hosted by the Ministry of
for real-time monitoring system in automotive manufacturing,’’ Sensors, Research, Technology, and Higher Education, Indonesia, in August 2019,
vol. 18, no. 9, p. 2946, Sep. 2018, doi: 10.3390/s18092946. to make contributions on accelerating the Indonesian national development.
[43] M. Syafrudin, N. Fitriyani, G. Alfian, and J. Rhee, ‘‘An affordable fast early He was recognized for Excellence in research, teaching, and outreach. He has
warning system for edge computing in assembly line,’’ Appl. Sci., vol. 9, published numerous research articles in several international peer-reviewed
no. 1, p. 84, Dec. 2018, doi: 10.3390/app9010084.
journals, including IEEE ACCESS, Food Control, Sensors, Applied Sciences,
[44] G. Lemaitre, F. Nogueira, and C. K. Aridas, ‘‘Imbalanced-learn: A Python
toolbox to tackle the curse of imbalanced datasets in machine learning,’’ Asia Pacific Journal of Marketing and Logistics, and Sustainability. His
J. Mach. Learn. Res., vol. 18, no. 1, pp. 559–563, Jan. 2017. research interests include industrial artificial intelligence, machine learning,
[45] T. Chen and C. Guestrin, ‘‘XGBoost: A scalable tree boosting sys- information systems, edge-computing, the Internet of Things, big data, health
tem,’’ in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery informatics, and smart factory ranging from theory to design and implemen-
Data Mining, San Francisco, CA, USA, Aug. 2016, pp. 785–794, doi: tation. He serves as a Reviewer Board Members for Sensors and Algorithms
10.1145/2939672.2939785. (MDPI) and a Review Editor for the IoT and Sensor Networks (Frontiers in
[46] C. Marrocco, R. P. W. Duin, and F. Tortorella, ‘‘Maximizing the Communications and Networks).
area under the ROC curve by pairwise feature combination,’’ Pattern
Recognit., vol. 41, no. 6, pp. 1961–1974, Jun. 2008, doi: 10.1016/
j.patcog.2007.11.017.
[47] K.-A. Toh, J. Kim, and S. Lee, ‘‘Maximizing area under ROC curve for bio-
metric scores fusion,’’ Pattern Recognit., vol. 41, no. 11, pp. 3373–3392,
Nov. 2008, doi: 10.1016/j.patcog.2008.04.002.
[48] S. H. Jee et al., ‘‘A coronary heart disease prediction model: The korean GANJAR ALFIAN (Member, IEEE) received the
heart study,’’ BMJ Open, vol. 4, no. 5, May 2014, Art. no. e005025, doi: B.Eng. degree from the Department of Informat-
10.1136/bmjopen-2014-005025. ics Engineering, Universitas Islam Negeri Sunan
[49] G. T. Reddy, M. P. K. Reddy, K. Lakshmanna, D. S. Rajput, R. Kaluri, Kalijaga, Yogyakarta, Indonesia, in 2009, and the
and G. Srivastava, ‘‘Hybrid genetic algorithm and a fuzzy logic classifier M.Eng. and Dr.Eng. degrees from the Department
for heart disease diagnosis,’’ Evol. Intell., vol. 13, no. 2, pp. 185–196, of Industrial and Systems Engineering, Dongguk
Nov. 2019, doi: 10.1007/s12065-019-00327-1. University, Seoul, South Korea, in 2012 and 2016,
[50] B. R. Kirkwood, J. A. C. Sterne, and B. R. Kirkwood, Essential Medical respectively. He has been an Assistant Profes-
Statistics, 2nd ed. Malden, MA, USA: Blackwell Science, 2003. sor with the Industrial AI Research Center, Nano
[51] M. Xu, D. Fralick, J. Z. Zheng, B. Wang, X. M. Tu, and C. Feng, ‘‘The dif-
Information Technology Academy, Dongguk Uni-
ferences and similarities between two-sample T-test and paired T-test,’’
versity, since 2016. In July 2017, he was a short-term Visiting Researcher
Shanghai Arch, Psychiatry, vol. 29, no. 3, pp. 184–188, Jun. 2017, doi:
10.11919/j.issn.1002-0829.217070. with the VSB-Technical University of Ostrava, Czech Republic. He was
[52] G. Alfian, M. Syafrudin, M. Ijaz, M. Syaekhoni, N. Fitriyani, and J. Rhee, recognized for Excellence in research, teaching, and outreach. He has
‘‘A personalized healthcare monitoring system for diabetic patients by uti- published numerous research articles in several international peer-reviewed
lizing BLE-based sensors and real-time data processing,’’ Sensors, vol. 18, journals, including Computers and Industrial Engineering, Journal of Food
no. 7, p. 2183, Jul. 2018, doi: 10.3390/s18072183. Engineering, Journal of Public Transportation, IEEE ACCESS, Food Control,
[53] N. L. Fitriyani, M. Syafrudin, G. Alfian, and J. Rhee, ‘‘Development of Sensors, Applied Sciences, Asia Pacific Journal of Marketing and Logistics,
disease prediction model based on ensemble learning approach for diabetes and Sustainability. His research interests include machine learning, deep
and hypertension,’’ IEEE Access, vol. 7, pp. 144777–144789, 2019, doi: learning, RFID, the Internet of Things, big data, health informatics, and
10.1109/ACCESS.2019.2945129. simulation and car sharing service. He was a recipient of the International
Conference on Science and Technology (ICST) Best Paper Award, in 2019.