生成对抗网络在医学图像计算上的进展与展望
Review and Prospects for Generative Adversarial Networks on Medical Image Computation

DOI: 10.12677/CSA.2021.117200, PP. 1949-1961

Keywords: 生成对抗网络，医学图像计算，深度学习
Generative Adversarial Networks, Medical Image Computation, Deep Learning

Full-Text   Cite this paper   Add to My Lib

Abstract:

生成对抗网络中生成器和判别器进行博弈的对抗训练方法在计算机视觉任务中引起了大量关注。生成器的参数更新不是直接来自真实数据样本，而是依据判别器的真伪和类别判别，从而具有生成媲美真实图像的能力。此外，生成对抗网络的对抗训练方式具有半监督/无监督训练特性，非常适合应用于医学图像计算领域，用以解决医学图像数据量少、质量低的缺陷。本文从不同角度对基于生成对抗网络的医学图像计算(医学图像合成、超分辨率重建和辅助诊断)的研究进展进行了回顾，并从模型设计、性能表现等方面对相关工作进行了概述和分析。最后，对生成对抗网络在该领域面临的挑战及潜在应用进行了展望。
In generator adversarial networks (GANs), the adversarial training mechanism between the generator and discriminator has attracted lots of attention in computer vision community. The update of the parameters in generator depends on the decision of the discriminator rather than the ground truth images, so the generator is able to synthesize more plausible images. In addition, the adversarial training mechanism makes GANs suitable for the semi-supervised/unsupervised training. This characteristic has been proven useful to address the problem that the medical images are limited and the quality is poor in the field of medical image computation. This paper reviews the researches of medical image computation (medical image synthesis, super-resolution and computer aided diagnosis) based on generative adversarial networks from different perspectives, and analyzes the related works from the aspects of model architecture and performance. Finally, the challenges and potential applications of generative adversarial networks in this field are presented.

References

[1]	Haris, M., Shakhnarovich, G. and Ukita, N. (2018) Deep Back-Projection Networks for Super-Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 1664-1673. https://doi.org/10.1109/CVPR.2018.00179
[2]	You, C., Li, G., Zhang, Y., et al. (2019) CT Super-Resolution GAN Constrained by the Identical, Residual, and Cycle Learning Ensemble (GAN-CIRCLE). IEEE Transactions on Medical Imaging, 39, 188-203. https://doi.org/10.1109/TMI.2019.2922960
[3]	Song, T.A., Chowdhury, S.R., Yang, F., et al. (2020) PET Image Super-Resolution Using Generative Adversarial Networks. Neural Networks, 125, 83-91. https://doi.org/10.1016/j.neunet.2020.01.029
[4]	Zhu, J., Yang, G. and Lio, P. (2019) How Can We Make Gan Perform Better in Single Medical Image Super-Resolution? A Lesion Focused Multi-Scale Approach. 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), Venice, 8-11 April 2019, 1669-1673. https://doi.org/10.1109/ISBI.2019.8759517
[5]	Huang, X., Zhang, Q., Wang, G., et al. (2019) Medical Image Super-Resolution Based on the Generative Adversarial Network. Chinese Intelligent Systems Conference, Haikou, 26-27 October 2019, 243-253. https://doi.org/10.1007/978-981-32-9686-2_29
[6]	Morrone, M.C. and Owens, R.A. (1987) Feature Detection from Local Energy. Pattern Recognition Letters, 6, 303-313. https://doi.org/10.1016/0167-8655(87)90013-4
[7]	Lyu, Q., Shan, H. and Wang, G. (2020) MRI Su-per-Resolution with Ensemble Learning and Complementary Priors. IEEE Transactions on Computational Imaging, 6, 615-624. https://doi.org/10.1109/TCI.2020.2964201
[8]	Wang, S.Q., Li, X., Cui, J.L., et al. (2015) Prediction of Myelopathic Level in Cervical Spondylotic Myelopathy Using Diffusion Tensor Imaging. Journal of Magnetic Reso-nance Imaging, 41, 1682-1688. https://doi.org/10.1002/jmri.24709
[9]	Wang, S., Hu, Y., Shen, Y., et al. (2018) Classification of Diffusion Ten-sor Metrics for the Diagnosis of a Myelopathic Cord Using Machine Learning. International Journal of Neural Systems, 28, Article ID: 1750036. https://doi.org/10.1142/S0129065717500368
[10]	Wang, S., Shen, Y., Shi, C., et al. (2018) Skeletal Maturity Recognition Using a Fully Automated System with Convolutional Neural Networks. IEEE Access, 6, 29979-29993. https://doi.org/10.1109/ACCESS.2018.2843392
[11]	Wang, S., Wang, X., Shen, Y., et al. (2020) An Ensem-ble-Based Densely-Connected Deep Learning System for Assessment of Skeletal Maturity. IEEE Transactions on Sys-tems, Man, and Cybernetics: Systems, 1-12. https://doi.org/10.1109/TSMC.2020.2997852
[12]	Wang, S., Shen, Y., Chen ,W., et al. (2017) Automatic Recog-nition of Mild Cognitive Impairment from Mri Images Using Expedited Convolutional Neural Networks. International Conference on Artificial Neural Networks, Alghero, 11-14 September 2017, 373-380. https://doi.org/10.1007/978-3-319-68600-4_43
[13]	Wang, S., Wang, H., Shen, Y., et al. (2018) Automatic Recognition of Mild Cognitive Impairment and Alzheimers Disease Using Ensemble Based 3D Densely Connected Convolutional Networks. 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, 17-20 December 2018, 517-523. https://doi.org/10.1109/ICMLA.2018.00083
[14]	Lei, B., Yang, M., Yang, P., et al. (2020) Deep and Joint Learn-ing of Longitudinal Data for Alzheimer’s Disease Prediction. Pattern Recognition, 102, Article ID: 107247. https://doi.org/10.1016/j.patcog.2020.107247
[15]	Wang, S., Wang, H., Cheung, A.C., et al. (2020) Ensemble of 3D Densely Connected Convolutional Network for Diagnosis of Mild Cognitive Impairment and Alzheimer’s Disease. Deep Learning Applications, 1098, 53-73. https://doi.org/10.1007/978-981-15-1816-4_4
[16]	Yu, S., Wang, S., Xiao, X., et al. (2020) Multi-Scale Enhanced Graph Convolutional Network for Early Mild Cognitive Impairment Detection. International Conference on Medical Image Computing and Computer-Assisted Intervention, Lima, 4-8 October 2020, 228-237. https://doi.org/10.1007/978-3-030-59728-3_23
[17]	Yu, W., Lei, B., Ng, M.K., et al. (2021) Tensorizing GAN with High-Order Pooling for Alzheimer’s Disease Assessment. IEEE Transactions on Neural Networks and Learning Systems, 1-15. https://doi.org/10.1109/TNNLS.2021.3063516
[18]	Lei, B., Xia, Z., Jiang, F., et al. (2020) Skin Lesion Segmenta-tion via Generative Adversarial Networks with Dual Discriminators. Medical Image Analysis, 64, Article ID: 101716. https://doi.org/10.1016/j.media.2020.101716
[19]	Burgos, N., Cardoso, M.J., Thielemans, K., et al. (2014) Attenuation Correction Synthesis for Hybrid PET-MR Scanners: Application to Brain Studies. IEEE transactions on medical imaging, 33, 2332-2341. https://doi.org/10.1109/TMI.2014.2340135
[20]	Wang, S., Wang, X., Hu, Y., et al. (2020) Diabetic Retinopathy Diagnosis Using Multichannel Generative Adversarial Network with Semisupervision. IEEE Transactions on Automa-tion Science and Engineering, 18, 574-585. https://doi.org/10.1109/TASE.2020.2981637
[21]	Zhang, R., Isola, P., Efros, A.A., et al. (2018) The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 586-595. https://doi.org/10.1109/CVPR.2018.00068
[22]	Odena, A., Olah, C. and Shlens, J. (2017) Conditional Image Synthesis with Auxiliary Classifier Gans. Proceedings of the 34th International Conference on Machine Learning, 70, 2642-2651.
[23]	Heusel, M., Ramsauer, H., Unterthiner, T., et al. (2017) Gans Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, Long Beach, 4-9 De-cember 2017, 6626-6637.
[24]	Karras, T., Laine, S. and Aila, T. (2019) A Style-Based Generator Architecture for Gen-erative Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 4401-4410. https://doi.org/10.1109/CVPR.2019.00453
[25]	Jog, A., Carass, A. and Prince, J.L. (2014) Improving Magnetic Resonance Resolution with Supervised Learning. 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI), Beijing, 29 April-2 May 2014, 987-990. https://doi.org/10.1109/ISBI.2014.6868038
[26]	Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) Imagenet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, Nevada, 3-6 December 2012, 1097-1105.
[27]	Zeng, D., Wang, S., Shen, Y., et al. (2017) A GA-Based Feature Selection and Parameter Optimization for Support Tucker Machine. Procedia Computer Science, 111, 17-23. https://doi.org/10.1016/j.procs.2017.06.004
[28]	Li, R., Zhang, W., Suk, H.I., et al. (2014) Deep Learning Based Imaging Data Completion for Improved Brain Disease Diagnosis. International Conference on Medical Image Compu-ting and Computer-Assisted Intervention, Boston, 14-18 September 2014, 305-312. https://doi.org/10.1007/978-3-319-10443-0_39
[29]	Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al. (2014) Generative Adversarial Nets. Advances in Neural Information Processing Systems, Montreal, 8-13 December 2014, 2672-2680.
[30]	Mo, L.F. and Wang, S.Q. (2009) A Variational Approach to Nonlinear Two-Point Boundary Value Problems. Nonlinear Analysis: Theory, Methods & Applications, 71, e834-e838. https://doi.org/10.1016/j.na.2008.12.006
[31]	Wang, S.Q. (2009) A Variational Approach to Nonlinear Two-Point Boundary Value Problems. Computers & Mathematics with Applications, 58, 2452-2455. https://doi.org/10.1016/j.camwa.2009.03.050
[32]	Wang, S.Q. and He, J.H. (2008) Variational Iteration Method for a Nonlinear Reaction-Diffusion Process. International Journal of Chemical Reactor Engineering, 6, A37. https://doi.org/10.2202/1542-6580.1630
[33]	Kingma, D.P. and Welling, M. (2013) Auto-Encoding Variational Bayes. arXiv Preprint, arXiv:1312.6114.
[34]	Park, T., Liu, M.Y., Wang, T.C., et al. (2019) Semantic Image Synthesis with Spatially-Adaptive Normalization. Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 2337-2346. https://doi.org/10.1109/CVPR.2019.00244
[35]	王坤峰, 苟超, 段艳杰, 等. 生成式对抗网络GAN的研究进展与展望[J]. 自动化学报, 2017, 43(3): 321-332.
[36]	Karras, T., Aila, T., Laine, S., et al. (2017) Progressive Growing of Gans for Improved Quality, Stability, and Variation. arXiv Preprint, arXiv:1710.10196.
[37]	Spampinato, C., Palazzo, S., D’Oro, P., et al. (2019) Adversarial Framework for Unsupervised Learning of Motion Dynamics in Videos. Internation-al Journal of Computer Vision, 128, 1378-1397.
[38]	Kim, T., Cha, M., Kim, H., et al. (2017) Learning to Discover Cross-Domain Relations with Generative Adversarial Networks. Proceedings of the 34th International Conference on Machine Learning, 70, 1857-1865.
[39]	Zhang, Y., Gan, Z. and Carin, L. (2016) Generating Text via Adversarial Training. NIPS Workshop on Adversarial Training, 21, 21-32.
[40]	Yu, L., Zhang, W., Wang, J., et al. (2017) Seqgan: Sequence Generative Adversarial Nets with Policy Gradient. Thirty-First AAAI Conference on Artificial Intelligence, 31, 2852-2858.
[41]	Radford, A., Metz, L. and Chintala, S. (2015) Unsupervised Representation Learning with Deep Con-volutional Generative Adversarial Networks. arXiv Preprint, arXiv:1511.06434.
[42]	Arjovsky, M., Chintala, S. and Bottou, L. (2017) Wasserstein Gan. arXiv Preprint, arXiv:1701.07875.
[43]	Isola, P., Zhu, J.Y., Zhou, T., et al. (2017) Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 1125-1134. https://doi.org/10.1109/CVPR.2017.632
[44]	Zhu, J.Y., Park, T., Isola, P., et al. (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. Proceedings of the IEEE International Conference on Com-puter Vision, Venice, 22-29 October 2017, 2223-2232. https://doi.org/10.1109/ICCV.2017.244
[45]	Frid-Adar, M., Diamant, I., Klang, E., et al. (2018) GAN-Based Synthetic Medical Image Augmentation for Increased CNN Perfor-mance in Liver Lesion Classification. Neurocomputing, 321, 321-331. https://doi.org/10.1016/j.neucom.2018.09.013
[46]	Bermudez, C., Plassard, A.J., Davis, L.T., et al. (2018) Learn-ing Implicit Brain MRI Manifolds with Deep Learning. Medical Imaging 2018: Image Processing, Houston. https://doi.org/10.1117/12.2293515
[47]	Chuquicusma, M.J.M., Hussein, S., Burt, J., et al. (2018) How to Fool Radiologists with Generative Adversarial Networks? A Visual Turing Test for Lung Cancer Diagnosis. 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), Washington, 4-7 April 2018, 240-244. https://doi.org/10.1109/ISBI.2018.8363564
[48]	Baur, C., Albarqouni, S. and Navab, N. (2018) Generating Highly Realistic Images of Skin Lesions with GANs. OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis. Springer, Cham, 260-267. https://doi.org/10.1007/978-3-030-01201-4_28
[49]	Kwon, G., Han, C. and Kim, D. (2019) Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks. International Conference on Medical Image Com-puting and Computer-Assisted Intervention, Shenzhen, 13-17 October 2019, 118-126. https://doi.org/10.1007/978-3-030-32248-9_14
[50]	Rosca, M., Lakshminarayanan, B., Warde-Farley, D., et al. (2017) Variational Approaches for Auto-Encoding Generative Adversarial Networks. arXiv Preprint, arXiv:1706.04987.
[51]	Zhang, T., Fu, H., Zhao, Y., et al. (2019) SkrGAN: Sketching-Rendering Unconditional Gen-erative Adversarial Networks for Medical Image Synthesis. International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, 13-17 October 2019, 777-785. https://doi.org/10.1007/978-3-030-32251-9_85
[52]	Hu, S., Shen, Y., Wang, S., et al. (2020) Brain MR to PET Synthesis via Bidirectional Generative Adversarial Network. International Conference on Medical Image Computing and Computer-Assisted Intervention, Lima, 4-8 October 2020, 698-707. https://doi.org/10.1007/978-3-030-59713-9_67
[53]	Hu, S., Yu, W., Chen, Z., et al. (2020) Medical Image Recon-struction Using Generative Adversarial Network for Alzheimer Disease Assessment with Class-Imbalance Problem. 2020 IEEE 6th International Conference on Computer and Communications (ICCC), Chengdu, 11-14 December 2020, 1323-1327. https://doi.org/10.1109/ICCC51575.2020.9344912
[54]	Hu, S., Yuan, J. and Wang, S. (2019) Cross-Modality Synthesis from MRI to PET Using Adversarial U-Net with Different Normalization. 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), Shenzhen, 22-24 November 2019, 1-5. https://doi.org/10.1109/ICMIPE47306.2019.9098219
[55]	Nie, D., Trullo, R., Lian, J., et al. (2018) Medical Image Synthesis with Deep Convolutional Adversarial Networks. IEEE Transactions on Biomedical Engineering, 65, 2720-2730. https://doi.org/10.1109/TBME.2018.2814538
[56]	Wei, W., Poirion, E., Bodini, B., et al. (2018) Learning Myelin Content in Multiple Sclerosis from Multimodal MRI through Adversarial Training. International Con-ference on Medical Image Computing and Computer-Assisted Intervention, Granada, 16-20 September 2018, 514-522. https://doi.org/10.1007/978-3-030-00931-1_59
[57]	Jack, Jr.C.R., Bernstein, M.A., Fox, N.C., et al. (2008) The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI Methods. Journal of Magnetic Resonance Imaging: An Offi-cial Journal of the International Society for Magnetic Resonance in Medicine, 27, 685-691.
[58]	Pan, Y., Liu, M., Lian, C., et al. (2018) Synthesizing Missing PET from MRI with Cycle-Consistent Generative Adversarial Networks for Alz-heimer’s Disease Diagnosis. International Conference on Medical Image Computing and Computer-Assisted Interven-tion, Granada, 16-20 September 2018, 455-463. https://doi.org/10.1007/978-3-030-00931-1_52
[59]	Pan, Y., Liu, M., Lian, C., et al. (2019) Disease-Image Spe-cific Generative Adversarial Network for Brain Disease Diagnosis with Incomplete Multi-Modal Neuroimages. Interna-tional Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, 13-17 October 2019, 137-145. https://doi.org/10.1007/978-3-030-32248-9_16
[60]	Yu, B., Zhou, L., Wang, L., et al. (2018) 3D cGAN Based Cross-Modality MR Image Synthesis for Brain Tumor Segmentation. 2018 IEEE 15th International Symposium on Bio-medical Imaging (ISBI 2018) Washington, 4-7 April 2018, 626-630. https://doi.org/10.1109/ISBI.2018.8363653
[61]	Shin, H.C., Tenenholtz, N.A., Rogers, J.K., et al. (2018) Medical Image Synthesis for Data Augmentation and Anonymization Using Generative Adversarial Networks. International Workshop on Simulation and Synthesis in Medical Imaging, Granada, 16 September 2018, 1-11. https://doi.org/10.1007/978-3-030-00536-8_1
[62]	Yang, H., Sun, J., Carass, A., et al. (2018) Unpaired Brain MR-to-CT Synthesis Using a Structure-Constrained CycleGAN. In: Stoyanov, D., et al., Eds., Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, Cham, 174-182. https://doi.org/10.1007/978-3-030-00889-5_20
[63]	Hiasa, Y., Otake, Y., Takao, M., et al. (2018) Cross-Modality Image Synthesis from Unpaired Data Using CycleGAN. In: Gooya, A., Goksel, O., Oguz, I., Burgos, N., Eds., Simula-tion and Synthesis in Medical Imaging, Springer, Cham, 31-41. https://doi.org/10.1007/978-3-030-00536-8_4
[64]	Zhang, Z., Yang, L. and Zheng, Y. (2018) Translating and Segmenting Multimodal Medical Volumes with cycle-And Shape-Consistency Generative Adversarial Network. Pro-ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 9242-9251. https://doi.org/10.1109/CVPR.2018.00963
[65]	Zhao, M., Wang, L., Chen, J., et al. (2018) Cranio-maxillofacial Bony Structures Segmentation from Mri with Deep-Supervision Adversarial Learning. International Con-ference on Medical Image Computing and Computer-Assisted Intervention, Granada, 16-20 September 2018, 720-727. https://doi.org/10.1007/978-3-030-00937-3_82
[66]	You, S., Liu, Y., Lei, B., et al. (2020) Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain. arXiv Preprint, arXiv:2011.04145,.
[67]	Mahapatra, D., Bozorgtabar, B. and Garnavi, R. (2019) Image Super-Resolution Using Progressive Generative Adversarial Networks for Medical Image Analysis. Computerized Medical Imaging and Graphics, 71, 30-39. https://doi.org/10.1016/j.compmedimag.2018.10.005
[68]	Zhao, M., Liu, X., Liu, H., et al. (2020) Su-per-Resolution of Cardiac Magnetic Resonance Images Using Laplacian Pyramid based on Generative Adversarial Net-works. Computerized Medical Imaging and Graphics, 80, Article ID: 101698. https://doi.org/10.1016/j.compmedimag.2020.101698
[69]	Burt, P. and Adelson, E. (1983) The Laplacian Pyramid as a Compact IMAGE code. IEEE Transactions on Communications, 31, 532-540. https://doi.org/10.1109/TCOM.1983.1095851
[70]	Bing, X., Zhang, W., Zheng, L., et al. (2019) Medical Image Super Resolution Using Improved Generative Adversarial Networks. IEEE Access, 7, 145030-145038. https://doi.org/10.1109/ACCESS.2019.2944862
[71]	Hu, J., Shen, L. and Sun, G. (2018) Squeeze-and-Excitation Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 7132-7141. https://doi.org/10.1109/CVPR.2018.00745
[72]	Lim, B., Son, S., Kim, H., et al. (2017) Enhanced Deep Residual Networks for Single Image Super-Resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, 21-26 July 2017, 136-144. https://doi.org/10.1109/CVPRW.2017.151
[73]	Ledig, C., Theis, L., Huszár, F., et al. (2017) Photo-Realistic Sin-gle Image Super-Resolution Using a Generative Adversarial Network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 4681-4690. https://doi.org/10.1109/CVPR.2017.19
[74]	Kim, J., Kwon Lee, J. and Mu Lee, K. (2016) Accurate Image Su-per-Resolution Using Very Deep Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 27-30 June 2016, 1646-1654. https://doi.org/10.1109/CVPR.2016.182

Full-Text