Abstract
Generative adversarial networks (GANs) are state-of-the-art neural network models
used to synthesise images and other data. GANs brought a considerable improvement
to the quality of synthetic data, quickly becoming the standard for data-generation
tasks. In this work, we summarise the applications of GANs in the field of cardiology,
including generation of realistic cardiac images, electrocardiography signals, and
synthetic electronic health records. The utility of GAN-generated data is discussed
with respect to research, clinical care, and academia. And we present illustrative
examples of our GAN-generated cardiac magnetic resonance and echocardiography images,
showing the evolution in image quality across 6 different models, which have become
almost indistinguishable from real images. Finally, we discuss future applications,
such as modality translation or patient trajectory modelling. Moreover, we discuss
the pending challenges that GANs need to overcome, namely, their training dynamics,
the medical fidelity or the data regulations and ethics questions, to become integrated
in cardiology workflows.
Résumé
Les réseaux antagonistes génératifs (RAG) sont des modèles de réseaux neuronaux de
pointe utilisés pour synthétiser des images et d’autres données. Les RAG ont permis
d’améliorer considérablement la qualité des données synthétiques, devenant ainsi rapidement
la norme en matière de génération de données. Dans cet article, nous résumons les
applications des RAG dans le domaine de la cardiologie, notamment la génération d’images
cardiaques, de signaux d’électrocardiographie et de dossiers médicaux électroniques
synthétiques réalistes. Nous abordons l’utilité des données fournies par les RAG sous
l’angle de la recherche, des soins cliniques et des travaux universitaires. Nous présentons
également des exemples d’images de résonance magnétique cardiaque et d’échocardiographie
que nous avons obtenues au moyen de RAG pour illustrer l’évolution de la qualité des
images à partir de six modèles différents, devenus quasiment impossibles à distinguer
des images réelles. Enfin, nous abordons les applications futures, dont le transfert
de modalités ou la modélisation de l'évolution de l'état clinique des patients. Par
ailleurs, nous nous penchons sur certains aspects problématiques – à savoir la dynamique
d’apprentissage, la fidélité médicale ou les questions de réglementation des données
et d’éthique – auxquels il convient de remédier pour permettre l’intégration des RAG
dans la pratique courante en cardiologie.
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Canadian Journal of CardiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
References
- Generative adversarial nets.in: Ghahramani Z. Welling M. Cortes C. NIPS’14: Proceedings of the 27th International Conference on Neural Information Processing Systems, Montréal, December 8-13, 2014. MIT Press, 2014: 2672-2680
- Anonymization through data synthesis using generative adversarial networks (ADS-GAN).IEEE J Biomed Health Inform. 2020; 24: 2378-2388
- A never-ending stream of AI art goes up for auction. The Verge; March 5, 2019.(Available at:)https://www.theverge.com/2019/3/5/18251267/ai-art-gans-mario-klingemann-auction-sothebys-technologyDate accessed: October 26, 2021
- Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved?.IEEE Trans Med Imaging. 2018; 37: 2514-2525
- Deep learning for segmentation using an open large-scale dataset in 2D echocardiography.IEEE Trans Med Imaging. 2019; 38: 2198-2210
Radford A, Metz L, Chintala S. Unsupervised representation learning with deep convolutional generative adversarial networks. In: Bengio Y, LeCun Y, eds. 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. arXiv:1511.06434.
- Least squares generative adversarial networks.in: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 2813-2821
Arjovsky M, Chintala M, Bottou L. Wasserstein generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning, July 2017. Proc Mach Learn Res 70:214-23.
Lim JH, Ye JC. Geometric GAN. arXiv:1705.02894.
Karras T, Laine S, Aittala M, et al. Analyzing and improving the image quality of StyleGAN. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020:8107-16.
Park T, Liu M-Y, Wang T-C, Zhu J-Y. Semantic image synthesis with spatially-adaptive normalization. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019:2332-41.
- Generative adversarial network in medical imaging: a review.Med Image Anal. 2019; 58: 101552
- XCAT-GAN for synthesizing 3D consistent labeled cardiac MR images on anatomically variable XCAT phantoms.in: Martel A.L. Abolmaesumi P. Stoyanov D. Medical Image Computing and Computer Assisted Intervention—MICCAI 2020. 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV. Springer, 2020: 128-137
- 4D XCAT phantom for multimodality imaging research.Med Phys. 2010; 37: 4902-4915
Skandarani Y, Painchaud N, Jodoin P-M, Lalande A. On the effectiveness of GAN generated cardiac MRIs for segmentation. arXiv:2005.09026.
- Translating and segmenting multimodal medical volumes with cycle- and shape-consistency generative adversarial network.in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 9242-9251
- Unpaired image-to-image translation using cycle-consistent adversarial networks.in: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 2242-2251
- Generative invertible networks (GIN): pathophysiology-interpretable feature mapping and virtual patient generation.in: Frangi A.F. Schnabel J.A. Davatzikos C. Alberola-López C. Fichtinger G. Medical Image Computing and Computer Assisted Intervention—MICCAI 2018. 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I. Springer, 2018: 537-545
- Super-resolution of cardiac MR cine imaging using conditional GANs and unsupervised transfer learning.Med Image Anal. 2021; 71: 102037
- The optimal tetralogy of Fallot repair using generative adversarial networks.Front Physiol. 2021; 12: 613330
- Electrocardiogram generation with a bidirectional LSTM-CNN generative adversarial network.Sci Rep. 2019; 9: 6734
- Generating electrocardiogram signals by deep learning.Neurocomputing. 2020; 404: 122-136
Sarkar P, Etemad A. CardioGAN: attentive generative adversarial network with dual discriminators for synthesis of ECG from PPG. arXiv:2010.00104.
- A new ECG denoising framework using generative adversarial network.IEEE/ACM Trans Comput Biol Bioinform. 2021; 18: 759-764
Georges-Filteau J, Cirillo E. Synthetic observational health data with GANs: from slow adoption to a boom in medical research and ultimately digital twins? Authorea, November 16, 2020.
- Synthesizing electronic health records using improved generative adversarial networks.J Am Med Inform Assoc. 2019; 26: 228-241
- Boosting deep learning risk prediction with generative adversarial networks for electronic health records.in: 2017 IEEE International Conference on Data Mining (ICDM). IEEE, 2017: 787-792
- Ensuring electronic medical record simulation through better training, modeling, and evaluation.J Am Med Inform Assoc. 2020; 27: 99-108
- A review: generative adversarial networks.in: 2019 14th IEEE Conference on Industrial Electronics and Applications (ICIEA). IEEE, 2019: 505-510
- Image-to-image translation with conditional adversarial networks.in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017: 5967-5976
Karras T, Aila T, Laine S, Lehtinen J. Progressive growing of GANs for improved quality, stability, and variation. arXiv:1710.10196.
- Medical image synthesis for data augmentation and anonymization using generative adversarial networks.in: Gooya A. Goksel O. Oguz I. Burgos N. Simulation and Synthesis in Medical Imaging. Third International Workshop, SASHIMI 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings. Springer, 2018: 1-11
- ScarGAN: chained generative adversarial networks to simulate pathological tissue on cardiovascular MR scans.in: Stoyanov D. Taylor Z. Carneiro G. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings. Springer, 2018: 343-350
- Utility of deep learning networks for the generation of artificial cardiac magnetic resonance images in congenital heart disease.BMC Med Imaging. 2020; 20: 113
- Improved techniques for training GANs.in: Lee D.D. von Luxburg U. Garnett R. Sugiyama M. Guyon I. NIPS’16: Proceedings of the 30th International Conference on Neural Information Processing Systems, December 5-10, 2016. Curran Associates, Barcelona. Red Hook, NY2016: 2234-2242
- GANs trained by a two time-scale update rule converge to a local Nash equilibrium.in: von Luxburg U. Guyon I. Bengio S. Wallach H. Fergus R. NIPS’17: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, December 4-9, 2017. Curran Associates, Red Hook, NY2017: 6629-6640
Bińkowski M, Sutherland DJ, Arbel M, Gretton A. Demystifying MMD GANs. arXiv1801.01401.
Article info
Publication history
Published online: November 12, 2021
Accepted:
November 8,
2021
Received:
September 17,
2021
Footnotes
See page 202 for disclosure information.
Identification
Copyright
© 2021 Canadian Cardiovascular Society. Published by Elsevier Inc. All rights reserved.
