Advertisement
Canadian Journal of Cardiology

Evaluation of Machine Learning Algorithms for Predicting Readmission After Acute Myocardial Infarction Using Routinely Collected Clinical Data

Published:October 25, 2019DOI:https://doi.org/10.1016/j.cjca.2019.10.023

      Abstract

      Background

      The ability to predict readmission accurately after hospitalization for acute myocardial infarction (AMI) is limited in current statistical models. Machine-learning (ML) methods have shown improved predictive ability in various clinical contexts, but their utility in predicting readmission after hospitalization for AMI is unknown.

      Methods

      Using detailed clinical information collected from patients hospitalized with AMI, we evaluated 6 ML algorithms (logistic regression, naïve Bayes, support vector machines, random forest, gradient boosting, and deep neural networks) to predict readmission within 30 days and 1 year of discharge. A nested cross-validation approach was used to develop and test models. We used C-statistics to compare discriminatory capacity, whereas the Brier score was used to indicate overall model performance. Model calibration was assessed using calibration plots.

      Results

      The 30-day readmission rate was 16.3%, whereas the 1-year readmission rate was 45.1%. For 30-day readmission, the discriminative ability for the ML models was modest (C-statistic 0.641; 95% confidence interval (CI), 0.621-0.662 for gradient boosting) and did not outperform previously reported methods. For 1-year readmission, different ML models showed moderate performance, with C-statistics around 0.72. Despite modest discriminatory capabilities, the observed readmission rates were markedly higher in the tenth decile of predicted risk compared with the first decile of predicted risk for both 30-day and 1-year readmission.

      Conclusions

      Despite including detailed clinical information and evaluating various ML methods, these models did not have better discriminatory ability to predict readmission outcomes compared with previously reported methods.

      Résumé

      Contexte

      Les modèles statistiques actuels ne permettent pas de prédire avec exactitude la réadmission après une hospitalisation pour cause d’infarctus aigu du myocarde (IAM). Les méthodes de prédiction faisant appel à l’apprentissage automatique ont été associées à une amélioration de la capacité de prédiction dans divers contextes cliniques, mais leur utilité pour prédire la réadmission après une hospitalisation pour cause d’IAM demeure inconnue.

      Méthodologie

      À l’aide de données cliniques détaillées recueillies auprès de patients hospitalisés pour un IAM, nous avons évalué six algorithmes d’apprentissage automatique (régression logistique, classification naïve bayésienne, machine à vecteurs de support, forêt aléatoire, boosting par descente de gradient fonctionnelle et réseaux neuronaux d’apprentissage profond) pour prédire la réadmission dans les 30 jours et dans l’année suivant la sortie de l’hôpital. Les modèles ont été mis au point et testés à l’aide d’une approche de validation croisée imbriquée. Nous avons utilisé la statistique C pour comparer la capacité de discrimination des différents modèles, et le score de Brier pour en chiffrer le rendement global. Le calage des modèles a été évalué au moyen de courbes d’étalonnage.

      Résultats

      Le taux de réadmission à 30 jours était de 16,3 %, tandis que le taux de réadmission à 1 an était de 45,1 %. Dans le cas de la réadmission à 30 jours, la capacité de discrimination des modèles d’apprentissage automatique était modeste (statistique C : 0,641; intervalle de confiance [IC] à 95 % : 0,621-0,662 pour le boosting par descente de gradient fonctionnelle) et n’était pas supérieure à celle des méthodes déjà utilisées. Dans le cas de la réadmission à 1 an, différents modèles d’apprentissage automatique se sont révélés modérément efficaces, la statistique C se chiffrant à environ 0,72. En dépit des modestes capacités de discrimination des différentes méthodes, les taux de réadmission observés étaient nettement plus élevés dans le dixième décile du risque prédit comparativement à ceux du premier décile, pour la réadmission à 30 jours comme pour la réadmission à 1 an.

      Conclusions

      Malgré le recours à des données cliniques détaillées et à différentes méthodes d’apprentissage automatique, les modèles évalués n’ont pas montré une capacité de discrimination supérieure à celle des méthodes déjà utilisées pour prédire la réadmission.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Canadian Journal of Cardiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Krumholz H.M.
        • Lin Z.
        • Keenan P.S.
        • et al.
        Relationship between hospital readmission and mortality rates for patients hospitalized with acute myocardial infarction, heart failure, or pneumonia.
        JAMA. 2013; 309: 587-593
        • Joynt K.E.
        • Jha A.K.
        Characteristics of hospitals receiving penalties under the Hospital Readmissions Reduction Program.
        JAMA. 2013; 309: 342-343
        • Kripalani S.
        • Theobald C.N.
        • Anctil B.
        • Vasilevskis E.E.
        Reducing hospital readmission rates: current strategies and future directions.
        Annu Rev Med. 2014; 65: 471-485
        • Dunlay S.M.
        • Pack Q.R.
        • Thomas R.J.
        • Killian J.M.
        • Roger V.L.
        Participation in cardiac rehabilitation, readmissions, and death after acute myocardial infarction.
        Am J Med. 2014; 127: 538-546
        • Feltner C.
        • Jones C.D.
        • Cene C.W.
        • et al.
        Transitional care interventions to prevent readmissions for persons with heart failure: a systematic review and meta-analysis.
        Ann Intern Med. 2014; 160: 774-784
        • van Walraven C.
        • Dhalla I.A.
        • Bell C.
        • et al.
        Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community.
        CMAJ. 2010; 182: 551-557
        • Zhou H.
        • Della P.R.
        • Roberts P.
        • Goh L.
        • Dhaliwal S.S.
        Utility of models to predict 28-day or 30-day unplanned hospital readmissions: an updated systematic review.
        BMJ Open. 2016; 6e011060
        • Wang H.
        • Robinson R.D.
        • Johnson C.
        • et al.
        Using the LACE index to predict hospital readmissions in congestive heart failure patients.
        BMC Cardiovasc Disord. 2014; 14: 97
        • Rana S.
        • Tran T.
        • Luo W.
        • Phung D.
        • Kennedy R.L.
        • Venkatesh S.
        Predicting unplanned readmission after myocardial infarction from routinely collected administrative hospital data.
        Aust Health Rev. 2014; 38: 377-382
        • Krumholz H.M.
        • Lin Z.
        • Drye E.E.
        • et al.
        An administrative claims measure suitable for profiling hospital performance based on 30-day all-cause readmission rates among patients with acute myocardial infarction.
        Circ Cardiovasc Qual Outcomes. 2011; 4: 243-252
        • Yu S.
        • Farooq F.
        • van Esbroeck A.
        • Fung G.
        • Anand V.
        • Krishnapuram B.
        Predicting readmission risk with institution-specific prediction models.
        Artif Intell Med. 2015; 65: 89-96
        • Smith L.N.
        • Makam A.N.
        • Darden D.
        • et al.
        Acute myocardial infarction readmission risk prediction models: a systematic review of model performance.
        Circ Cardiovasc Qual Outcomes. 2018; 11e003885
        • Crown W.H.
        Potential application of machine learning in health outcomes research and some statistical cautions.
        Value Health. 2015; 18: 137-140
        • Weng S.F.
        • Reps J.
        • Kai J.
        • Garibaldi J.M.
        • Qureshi N.
        Can machine-learning improve cardiovascular risk prediction using routine clinical data?.
        PLoS One. 2017; 12e0174944
        • Tu J.V.
        • Donovan L.R.
        • Lee D.S.
        • et al.
        Effectiveness of public report cards for improving the quality of cardiac care: the EFFECT study: a randomized trial.
        JAMA. 2009; 302: 2330-2337
        • Murphy K.P.
        Machine Learning: A Probabilistic Perspective.
        MIT Press, Cambridge, MA2012: 82-83
        • Cortes C.
        • Vapnik V.
        Support-sector networks.
        Machine Learning. 1995; 20: 273-297
        • Breiman L.
        Random forests.
        Machine Learning. 2001; 45: 5-32
        • Friedman J.H.
        Greedy function approximation: a gradient boosting machine.
        Ann Stat. 2001; 29: 1189-1232
        • Rumelhart D.E.
        • Hinton G.E.
        • Williams R.J.
        Learning representations by back-propagating errors.
        Nature. 1986; 323: 533-536
        • Pedregosa F.
        • Varoquaux G.
        • Gramfort A.
        • et al.
        Scikit-learn: Machine learning in python.
        J Machine Learning Res. 2011; 23: 2825-2830
      1. Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016; San Francisco, California.

        • Chollet F.
        Keras.
        (Updated 2015)
        • Steyerberg E.W.
        • Vickers A.J.
        • Cook N.R.
        • et al.
        Assessing the performance of prediction models: a framework for traditional and novel measures.
        Epidemiology. 2010; 21: 128-138
        • Khera R.
        • Jain S.
        • Pandey A.
        • et al.
        Comparison of readmission rates after acute myocardial infarction in 3 patient age groups (18 to 44, 45 to 64, and ≥65 years) in the United States.
        Am J Cardiol. 2017; 120: 1761-1767
        • Meddings J.
        • Reichert H.
        • Smith S.N.
        • et al.
        The impact of disability and social determinants of health on condition-specific readmissions beyond Medicare risk adjustments: a cohort study.
        J Gen Intern Med. 2017; 32: 71-80
        • Brown J.R.
        • Conley S.M.
        • Niles N.W.
        • II
        Predicting readmission or death after acute ST-elevation myocardial infarction.
        Clin Cardiol. 2013; 36: 570-575
        • McManus D.D.
        • Saczynski J.S.
        • Lessard D.
        • et al.
        Reliability of predicting early hospital readmission after discharge for an acute coronary syndrome using claims-based data.
        Am J Cardiol. 2016; 117: 501-507
        • Burke R.E.
        • Schnipper J.L.
        • Williams M.V.
        • et al.
        The HOSPITAL score predicts potentially preventable 30-day readmissions in conditions targeted by the hospital readmissions reduction program.
        Med Care. 2017; 55: 285-290
        • Joynt K.E.
        • Orav E.J.
        • Jha A.K.
        Thirty-day readmission rates for Medicare beneficiaries by race and site of care.
        JAMA. 2011; 305: 675-681
        • Kansagara D.
        • Englander H.
        • Salanitro A.
        • et al.
        Risk prediction models for hospital readmission: a systematic review.
        JAMA. 2011; 306: 1688-1698
        • Amarasingham R.
        • Moore B.J.
        • Tabak Y.P.
        • et al.
        An automated model to identify heart failure patients at risk for 30-day readmission or death using electronic medical record data.
        Med Care. 2010; 48: 981-988
        • Foraker R.E.
        • Rose K.M.
        • Suchindran C.M.
        • Chang P.P.
        • McNeill A.M.
        • Rosamond W.D.
        Socioeconomic status, Medicaid coverage, clinical comorbidity, and rehospitalization or death after an incident heart failure hospitalization: atherosclerosis risk in communities cohort (1987 to 2004).
        Circ Heart Fail. 2011; 4: 308-316
        • Herrin J.
        • St Andre J.
        • Kenward K.
        • Joshi M.S.
        • Audet A.M.
        • Hines S.C.
        Community factors and hospital readmission rates.
        Health Serv Res. 2015; 50: 20-39
        • Alter D.A.
        • Franklin B.
        • Ko D.T.
        • et al.
        socioeconomic status, functional recovery, and long-term mortality among patients surviving acute myocardial infarction.
        PLoS One. 2013; 8e65130
        • Singh S.
        • Lin Y.L.
        • Kuo Y.F.
        • Nattinger A.B.
        • Goodwin J.S.
        Variation in the risk of readmission among hospitals: the relative contribution of patient, hospital and inpatient provider characteristics.
        J Gen Intern Med. 2014; 29: 572-578
        • Rajkomar A.
        • Oren E.
        • Chen K.
        • et al.
        Scalable and accurate deep learning with electronic health records.
        NPJ Digital Med. 2018; 1: 18
        • Rennke S.
        • Nguyen O.K.
        • Shoeb M.H.
        • Magan Y.
        • Wachter R.M.
        • Ranji S.R.
        Hospital-initiated transitional care interventions as a patient safety strategy: a systematic review.
        Ann Intern Med. 2013; 158: 433-440
        • Wong G.C.
        • Welsford M.
        • Ainsworth C.
        • et al.
        2019 Canadian Cardiovascular Society/Canadian Association of Interventional Cardiology Guidelines on the Acute Management of ST-Elevation Myocardial Infarction: focused update on regionalization and reperfusion.
        Can J Cardiol. 2019; 35: 107-132