The Rise of Open-Sourced Machine Learning in Small and Imbalanced Datasets: Predicting In-Stent Restenosis

After the successful placement of a coronary stent and revascularization, in-stent restenosis (ISR) occurs in up to 12% of patients and represents the gradual failure of the stent by lumen renarrowing. Clinically, ISR is important to recognize because it is usually associated with recurrent angina symptoms, a higher risk for acute coronary syndrome, and increased mortality. The ability to accurately predict ISR would enable closer monitoring of those patients at higher risk of ISR or the consideration of alternate therapies. Accordingly, various risk prediction models have been developed to predict ISR, such as Prevention of Restenosis With Tranilast and Its Outcomes (PRESTO)-1, PRESTO-2, and the and Evaluation of Drug-Eluting Stents and Ischemic Events (EVENT) risk score (Table 1). However, these prediction models generally suffer from low predictive power: Areas under the receiver operating characteristic curve (AUC-ROCs) are 0.63 for PRESTO-1,² 0.63 for PRESTO-2, and 0.68 for EVENT. This presents an opportunity to develop models that more accurately predict the occurrence of ISR.

In recent years, there has been increasing interest in applying machine-learning algorithms toward various tasks in medicine, including risk prediction. Machine-learning algorithms offer several potential advantages over conventional statistical techniques used in risk prediction, such as improved performance as training data increases and the ability to model nonlinear relationships within the data. Such models often can accommodate a far larger number of predictors (also called “features”) but may require large dataset sizes to achieve improved predictive power over conventional risk-prediction models. Machine-learning algorithms also have limitations, which include less interpretability than general linear models, and may not offer performance improvements in settings of small dataset sizes or fewer numbers of predictive features.

In this issue of the Canadian Journal of Cardiology, Sampedro-Gómez et al. applied various machine-learning algorithms to develop risk models for the prediction of ISR. The authors sought to predict ISR in patients with ST-segment-elevation myocardial infarction (STEMI) undergoing percutaneous coronary intervention (PCI) within the Grupo de Análisis de la Cardiopatía Isquemica Aguda (GRACIA)-3 study population. The GRACIA-3 trial was a 2×2 randomized, open-label, multicenter trial that compared the efficacy of the paclitaxel-eluting stent with conventional bare-metal stents in patients with STEMI. ISR was assessed by core laboratory–validated coronary angiography at baseline and at 12-month follow-up in all patients. To develop machine learning–based risk prediction models, the authors used the 23 available case subjects with ISR and 240 control subjects in GRACIA-3, obtaining 68 demographic, clinical, and angiographic characteristics from each patient. A notable characteristic of the data used was that there were relatively low numbers of ISR events available (n = 23), and the total dataset size was also not very large (n = 263), both characteristics which might be less favourable to attempting machine-learning approaches. To address the smaller size of their dataset, the authors appropriately used an approach called k-fold cross-validation, which serves to provide more generalizable estimates of model performance and limit overfitting by splitting the dataset repeatedly into k-sized parts. Although this can be less ideal than maintaining a true hold-out subset of the data for model testing, k-fold cross-validation can be useful when the dataset is too small to realistically maintain a hold-out test subset. In addition, because the balance between those with and without ISR was nearly 1:10, this dataset was considered to be “imbalanced,” which can have implications regarding both model training and the assessment of model performance. To account for this dataset imbalance, the authors, again appropriately, examined the area under the precision-recall curve (AUC-PR) in addition to the more common AUC-ROC. Unlike, the AUC-ROC, which can be overly optimistic in the setting of imbalanced data, the AUC-PR is less affected by imbalanced data and paints a more complete picture of the predictive performance of the model in the minority class, which in this dataset represented the 23 cases of ISR (9% rate). Using both of these techniques, in part to address the limitation of having a lower number of ISR events and a smaller dataset size, the authors then developed risk-prediction models using 6 different machine-learning classifiers. They compared the performance of these machine learning–derived models against existing risk prediction models, including PRESTO-1, PRESTO-2, and EVENT scores applied to the GRACIA-3 population.

The best-performing model was developed using an extremely randomized trees (ERT) classifier, but it was not statistically different from the other machine-learning models used, except for the support vector classifier. The small dataset size may have contributed to the inability to detect differences between the various machine-learning classifiers. Compared with existing ISR prediction models, the ERT classifier achieved a higher AUC-PR of 0.46 than the other models applied to this study cohort (AUC-PRs of 0.31 for PRESTO-1, 0.27 for PRESTO-2, and 0.18 for EVENT). Again, likely because of the small dataset size, the confidence intervals of the AUC-PRs for all of these models were large, and the AUC-PR of the best-performing machine-learning model was not statistically significantly higher than those for PRESTO-1 or PRESTO-2. However, the ERT model appeared to outperform the EVENT risk score, and there was a trend toward improved discrimination compared with both PRESTO scores. It is noteworthy that despite the small dataset size and low numbers of events, the machine-learning models performed as well as they did. In our view, some of the greatest potential may lie in applying similar methods as shown in this work to similar prediction tasks with larger datasets and greater numbers of events.

In addition to performing ISR prediction, the most important predictors out of the 68 available predictorsc were identified during the feature selection process. The GRACIA-3 population looked at STEMI patients undergoing PCI, which is inherently different from the population of the PRESTO and EVENT studies, which recruited a majority of non-STEMI cases.^, Several key predictors identified by Sampedro-Gómez et al., such as ≥ 2 coronary vessel disease, post-PCI Thrombolysis in Myocardial Infarction flow, post-PCI thrombus, abnormal platelets count, and elevated total cholesterol, have not been consistently reported in previous models that looked at non-STEMI cases. Only diabetes has previously been used for ISR risk prediction in the PRESTO-1, PRESTO-2, and EVENT risk models.^, The other predictors highlighted in this study could represent novel predictors of post-STEMI PCI–associated ISR. It is particularly noteworthy to see that the strongest predictors of ISR in the ERT machine-learning model are supported by literature and consistent with our understanding of ISR pathophysiology, involving both neointimal proliferation and neoatherosclerosis. Future studies with larger numbers of events and larger datasets should examine the value of these features and, if consistent, these features could be considered for inclusion in future ISR risk-prediction models. Furthermore, the feature selection approach used in this study could be valuable in other settings to identify other novel predictors of ISR. Because the strongest predictors in this model are readily available in clinical practice, this model can be easily clinically applied to identify the subset of STEMI patients who are at risk for ISR after PCI, prompting more frequent follow-up to rule out the development of ISR.

The open-source code made available by the authors to accompany this article makes it possible for other institutions and investigators to develop similar machine-learning approaches to data analysis. This study provides an important proof-of-concept demonstration that machine-learning models can be used to develop effective risk-prediction models for ISR. Because the findings from this study were likely limited by the size of the available data, by open-sourcing the code, the authors have laid the groundwork to replicate their methods with the use of larger datasets or possibly datasets combined from multiple institutions or clinical trials. Using datasets of larger size as well as larger numbers of predictors would enable the community to truly leverage the strengths of machine-learning approaches. In addition, open-sourcing machine-learning models enables rapid confirmation, implementation, and iterative improvement, which should be a standard to which we all aspire, especially for machine-learning applications in medicine.

Despite the methodology and findings, there remain important limitations to the work presented by Sampedro-Gómez. The GRACIA-3 trial compared first-generation drug-eluting stents with bare-metal stents. Second-generation drug-eluting stents implanted during STEMI could have a different risk profile for ISR, and the model developed here might not apply to that population. In addition, the event rate was low (9%), and the trial lacked an external validation in an independent cohort. Until the latter is established, individualized risk of ISR predicted by this machine-learning model needs to be taken with a grain of salt because the model may suffer from overfitting and lack external generalizability. However, the authors used approaches such as cross-validation which alleviate some of these concerns.

In summary, Sampedro-Gómez et al. have described a process to apply machine-learning algorithms to develop different models predicting the 12-month risk of ISR in a STEMI population receiving first-generation drug-eluting stents or bare-metal stents. The best-performing model they report provides superior risk prediction for ISR compared with at least 1 previously published model and has the potential to be further improved if more data are available. Other risk-prediction efforts that use similarly small or imbalanced datasets might consider adopting a similar approach of cross-validation and examination of AUC-PRs. Future research, such as prospective validation of this score in other cohorts and evaluation of the reported novel predictors, will be crucial to determine the clinical implications of these findings.