Advertisement
Canadian Journal of Cardiology

Generative Adversarial Networks in Cardiology

Published:November 12, 2021DOI:https://doi.org/10.1016/j.cjca.2021.11.003

      Abstract

      Generative adversarial networks (GANs) are state-of-the-art neural network models used to synthesise images and other data. GANs brought a considerable improvement to the quality of synthetic data, quickly becoming the standard for data-generation tasks. In this work, we summarise the applications of GANs in the field of cardiology, including generation of realistic cardiac images, electrocardiography signals, and synthetic electronic health records. The utility of GAN-generated data is discussed with respect to research, clinical care, and academia. And we present illustrative examples of our GAN-generated cardiac magnetic resonance and echocardiography images, showing the evolution in image quality across 6 different models, which have become almost indistinguishable from real images. Finally, we discuss future applications, such as modality translation or patient trajectory modelling. Moreover, we discuss the pending challenges that GANs need to overcome, namely, their training dynamics, the medical fidelity or the data regulations and ethics questions, to become integrated in cardiology workflows.

      Résumé

      Les réseaux antagonistes génératifs (RAG) sont des modèles de réseaux neuronaux de pointe utilisés pour synthétiser des images et d’autres données. Les RAG ont permis d’améliorer considérablement la qualité des données synthétiques, devenant ainsi rapidement la norme en matière de génération de données. Dans cet article, nous résumons les applications des RAG dans le domaine de la cardiologie, notamment la génération d’images cardiaques, de signaux d’électrocardiographie et de dossiers médicaux électroniques synthétiques réalistes. Nous abordons l’utilité des données fournies par les RAG sous l’angle de la recherche, des soins cliniques et des travaux universitaires. Nous présentons également des exemples d’images de résonance magnétique cardiaque et d’échocardiographie que nous avons obtenues au moyen de RAG pour illustrer l’évolution de la qualité des images à partir de six modèles différents, devenus quasiment impossibles à distinguer des images réelles. Enfin, nous abordons les applications futures, dont le transfert de modalités ou la modélisation de l'évolution de l'état clinique des patients. Par ailleurs, nous nous penchons sur certains aspects problématiques – à savoir la dynamique d’apprentissage, la fidélité médicale ou les questions de réglementation des données et d’éthique – auxquels il convient de remédier pour permettre l’intégration des RAG dans la pratique courante en cardiologie.
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Canadian Journal of Cardiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Goodfellow I.
        • Pouget-Abadie J.
        • Mirza M.
        • et al.
        Generative adversarial nets.
        in: Ghahramani Z. Welling M. Cortes C. NIPS’14: Proceedings of the 27th International Conference on Neural Information Processing Systems, Montréal, December 8-13, 2014. MIT Press, 2014: 2672-2680
        • Yoon J.
        • Drumright L.N.
        • van der Schaar M.
        Anonymization through data synthesis using generative adversarial networks (ADS-GAN).
        IEEE J Biomed Health Inform. 2020; 24: 2378-2388
        • Vincent J.
        A never-ending stream of AI art goes up for auction. The Verge; March 5, 2019.
        (Available at:)
        • Bernard O.
        • Lalande A.
        • Zotti C.
        • et al.
        Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?.
        IEEE Trans Med Imaging. 2018; 37: 2514-2525
        • Leclerc S.
        • Smistad E.
        • Pedrosa J.
        • et al.
        Deep learning for segmentation using an open large-scale dataset in 2D echocardiography.
        IEEE Trans Med Imaging. 2019; 38: 2198-2210
      1. Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. In: Bengio Y, LeCun Y, eds. 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. arXiv:1511.06434.

        • Mao X.
        • Li Q.
        • Xie H.
        • et al.
        Least squares generative adversarial networks.
        in: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 2813-2821
      2. Arjovsky M, Chintala M, Bottou L. Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, July 2017. Proc Mach Learn Res 70:214-23.

      3. Lim JH, Ye JC. Geometric GAN. arXiv:1705.02894.

      4. Karras T, Laine S, Aittala M, et al. Analyzing and improving the image quality of StyleGAN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020:8107-16.

      5. Park T, Liu M-Y, Wang T-C, Zhu J-Y. Semantic image synthesis with spatially-adaptive normalization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019:2332-41.

        • Yi X.
        • Walia E.
        • Babyn P.
        Generative adversarial network in medical imaging: a review.
        Med Image Anal. 2019; 58: 101552
        • Amirrajab S.
        • Abbasi-Sureshjani S.
        • Al Khalil Y.
        • et al.
        XCAT-GAN for synthesizing 3D consistent labeled cardiac MR images on anatomically variable XCAT phantoms.
        in: Martel A.L. Abolmaesumi P. Stoyanov D. Medical Image Computing and Computer Assisted Intervention—MICCAI 2020. 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV. Springer, 2020: 128-137
        • Segars W.P.
        • Sturgeon G.
        • Mendonca S.
        • Grimes J.
        • Tsui B.M.W.
        4D XCAT phantom for multimodality imaging research.
        Med Phys. 2010; 37: 4902-4915
      6. Skandarani Y, Painchaud N, Jodoin P-M, Lalande A. On the effectiveness of GAN generated cardiac MRIs for segmentation. arXiv:2005.09026.

        • Zhang Z.
        • Yang L.
        • Zheng Y.
        Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network.
        in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 9242-9251
        • Zhu J.-Y.
        • Park T.
        • Isola P.
        • Efros A.A.
        Unpaired image-to-image translation using cycle-consistent adversarial networks.
        in: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 2242-2251
        • Chen J.
        • Xie Y.
        • Wang K.
        • et al.
        Generative invertible networks (GIN): pathophysiology-interpretable feature mapping and virtual patient generation.
        in: Frangi A.F. Schnabel J.A. Davatzikos C. Alberola-López C. Fichtinger G. Medical Image Computing and Computer Assisted Intervention—MICCAI 2018. 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I. Springer, 2018: 537-545
        • Xia Y.
        • Ravikumar N.
        • Greenwood J.P.
        • et al.
        Super-resolution of cardiac MR cine imaging using conditional GANs and unsupervised transfer learning.
        Med Image Anal. 2021; 71: 102037
        • Zhang G.
        • Mao Y.
        • Li M.
        • et al.
        The optimal tetralogy of Fallot repair using generative adversarial networks.
        Front Physiol. 2021; 12: 613330
        • Zhu F.
        • Ye F.
        • Fu Y.
        • Liu Q.
        • Shen B.
        Electrocardiogram generation with a bidirectional LSTM-CNN generative adversarial network.
        Sci Rep. 2019; 9: 6734
        • Wulan N.
        • Wang W.
        • Sun P.
        • et al.
        Generating electrocardiogram signals by deep learning.
        Neurocomputing. 2020; 404: 122-136
      7. Sarkar P, Etemad A. CardioGAN: attentive generative adversarial network with dual discriminators for synthesis of ECG from PPG. arXiv:2010.00104.

        • Singh P.
        • Pradhan G.
        A new ECG denoising framework using generative adversarial network.
        IEEE/ACM Trans Comput Biol Bioinform. 2021; 18: 759-764
      8. Georges-Filteau J, Cirillo E. Synthetic observational health data with GANs: from slow adoption to a boom in medical research and ultimately digital twins? Authorea, November 16, 2020.

        • Baowaly M.K.
        • Lin C.-C.
        • Liu C.-L.
        • Chen K.-T.
        Synthesizing electronic health records using improved generative adversarial networks.
        J Am Med Inform Assoc. 2019; 26: 228-241
        • Che Z.
        • Cheng Y.
        • Zhai S.
        • Sun Z.
        • Liu Y.
        Boosting deep learning risk prediction with generative adversarial networks for electronic health records.
        in: 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 2017: 787-792
        • Zhang Z.
        • Yan C.
        • Mesa D.A.
        • Sun J.
        • Malin B.A.
        Ensuring electronic medical record simulation through better training, modeling, and evaluation.
        J Am Med Inform Assoc. 2020; 27: 99-108
        • Gonog L.
        • Zhou Y.
        A review: generative adversarial networks.
        in: 2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA). IEEE, 2019: 505-510
        • Isola P.
        • Zhu J.-Y.
        • Zhou T.
        • Efros A.A.
        Image-to-image translation with conditional adversarial networks.
        in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 5967-5976
      9. Karras T, Aila T, Laine S, Lehtinen J. Progressive growing of GANs for improved quality, stability, and variation. arXiv:1710.10196.

        • Shin H.-C.
        • Tenenholtz N.A.
        • Rogers J.K.
        • et al.
        Medical image synthesis for data augmentation and anonymization using generative adversarial networks.
        in: Gooya A. Goksel O. Oguz I. Burgos N. Simulation and Synthesis in Medical Imaging. Third International Workshop, SASHIMI 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings. Springer, 2018: 1-11
        • Lau F.
        • Hendriks T.
        • Lieman-Sifry J.
        • Sall S.
        • Golden D.
        ScarGAN: chained generative adversarial networks to simulate pathological tissue on cardiovascular MR scans.
        in: Stoyanov D. Taylor Z. Carneiro G. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings. Springer, 2018: 343-350
        • Diller G.-P.
        • Vahle J.
        • Radke R.
        • et al.
        Utility of deep learning networks for the generation of artificial cardiac magnetic resonance images in congenital heart disease.
        BMC Med Imaging. 2020; 20: 113
        • Salimans T.
        • Goodfellow I.
        • Zaremba W.
        • et al.
        Improved techniques for training GANs.
        in: Lee D.D. von Luxburg U. Garnett R. Sugiyama M. Guyon I. NIPS’16: Proceedings of the 30th International Conference on Neural Information Processing Systems, December 5-10, 2016. Curran Associates, Barcelona. Red Hook, NY2016: 2234-2242
        • Heusel M.
        • Ramsauer H.
        • Unterthiner T.
        • Nessler B.
        • Hochreiter S.
        GANs trained by a two time-scale update rule converge to a local Nash equilibrium.
        in: von Luxburg U. Guyon I. Bengio S. Wallach H. Fergus R. NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, December 4-9, 2017. Curran Associates, Red Hook, NY2017: 6629-6640
      10. Bińkowski M, Sutherland DJ, Arbel M, Gretton A. Demystifying MMD GANs. arXiv1801.01401.