Using Cross Entropy as a Performance Metric for Quantifying Uncertainty in DNN Image Classifiers: An Application to Classification of Lung Cancer on CT Images
Cross entropy is a measure in machine
learning and deep learning that assesses the difference between predicted and
actual probability distributions. In this study, we propose cross entropy as a
performance evaluation metric for image classifier models and apply it to the
CT image classification of lung cancer. A convolutional neural network is
employed as the deep neural network (DNN) image classifier, with the residual
network (ResNet) 50 chosen as the DNN archi-tecture. The image data used
comprise a lung CT image set. Two classification models are built from datasets
with varying amounts of data, and lung cancer is categorized into four classes
using 10-fold cross-validation. Furthermore, we employ t-distributed stochastic
neighbor embedding to visually explain the data distribution after
classification. Experimental results demonstrate that cross en-tropy is a
highly useful metric for evaluating the reliability of image classifier models.
It is noted that for a more comprehensive evaluation of model perfor-mance,
combining with other evaluation metrics is considered essential.
References
[1]
Siegel, R.L., Miller, K.D., Wagle, N.S. and Jemal, A. (2023) Cancer Statistics, 2023. CA: A Cancer Journal for Clinicians, 73, 17-48. https://doi.org/10.3322/caac.21763
[2]
Xu, R., Lu, T., Wang, C., Li, Q., Peng, B., Zhao, J., et al. (2023) Single-Cell Data Analysis of Malignant Epithelial Cell Heterogeneity in Lung Adenocarcinoma for Patient Classification and Prognosis Prediction. Heliyon, 9, e20164. https://doi.org/10.1016/j.heliyon.2023.e20164
[3]
Niu, Z., Jin, R., Zhang, Y. and Li, H. (2022) Signaling Pathways and Targeted Therapies in Lung Squamous Cell Carcinoma: Mechanisms and Clinical Trials. Signal Transduction and Targeted Therapy, 7, 353. https://doi.org/10.1038/s41392-022-01200-x
[4]
Copin, M.-C. (2016) Carcinome à Grandes Cellules, Carcinome Lymphoepithelioma-Like, Carcinome NUT Large Cell Carcinoma, Lymphoepithelioma-Like Carcinoma, NUT Carcinoma. Annales de Pathologie, 36, 24-33. https://doi.org/10.1016/j.annpat.2015.11.006
[5]
The National Lung Screening Trial Research Team (2011) Reduced Lung-Cancer Mortality with Low-Dose Computed Tomographic Screening. The New England Journal of Medicine, 365, 395-409. https://doi.org/10.1056/NEJMoa1102873
[6]
Aberle, D.R., DeMello, S., Berg, C.D., Black, W.C., Brewer, B., Church, T.R., et al. (2013) Results of the Two Incidence Screenings in the National Lung Screening Trial. The New England Journal of Medicine, 369, 920-931. https://doi.org/10.1056/NEJMoa1208962
[7]
The National Lung Screening Trial Research Team (2013) Results of Initial Low-Dose Computed Tomographic Screening for Lung Cancer. The New England Journal of Medicine, 368, 1980-1991. https://doi.org/10.1056/NEJMoa1209120
[8]
Kramer, B.S., Berg, C.D., Aberle, D.R. and Prorok, P.C. (2011) Lung Cancer Screening with Low-Dose Helical CT: Results from the National Lung Screening Trial (NLST). Journal of Medical Screening, 18, 109-111. https://doi.org/10.1258/jms.2011.011055
[9]
Midthun, D.E. (2011) Screening for Lung Cancer. Clinics in Chest Medicine, 32, 659-668. https://doi.org/10.1016/j.ccm.2011.08.014
[10]
Goo, J.M. (2011) A Computer-Aided Diagnosis for Evaluating Lung Nodules on Chest CT: The Current Status and Perspective. Korean Journal of Radiology, 12, 145-155. https://doi.org/10.3348/kjr.2011.12.2.145
[11]
Suzuki, K. (2012) A Review of Computer-Aided Diagnosis in Thoracic and Colonic Imaging. Quantitative Imaging in Medicine and Surgery, 2, 163-176.
[12]
El-Baz, A., Beache, G.M., Gimel’farb, G., Suzuki, K., Okada, K., et al. (2013) Computer-Aided Diagnosis Systems for Lung Cancer: Challenges and Methodologies. International Journal of Biomedical Imaging, 2013, Article ID: 942353. https://doi.org/10.1155/2013/942353
[13]
Retico, A. (2013) Computer-Aided Detection for Pulmonary Nodule Identification: Improving the Radiologist’s Performance? Imaging in Medicine, 5, 249-263. https://doi.org/10.2217/iim.13.24
[14]
Firmino, M., Morais, A.H., Mendoca, R.M., Dantas, M.R., Hekis, H.R. and Valentim, R. (2014) Computer-Aided Detection System for Lung Cancer in Computed Tomography Scans: Review and Future Prospective. Biomedical Engineering Online, 13, 1-16. https://doi.org/10.1186/1475-925X-13-41
[15]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., et al. (2017) Attention Is All You Need.
[16]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., et al. (2020) An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.
[17]
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A. and Mougiakakou, S. (2016) Lung Pattern Classification for Interstitial Lung Diseases Using a Deep Convolutional Neural Network. IEEE Transactions on Medical Imaging, 35, 1207-1216. https://doi.org/10.1109/TMI.2016.2535865
[18]
Gao, M., Bagci, U., Lu, L., Wu, A., Buty, M., Shin, H.C., et al. (2018) Holistic Classification of CT Attenuation Patterns for Interstitial Lung Diseases via Deep Convolutional Neural Networks. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 6, 1-6. https://doi.org/10.1080/21681163.2015.1124249
[19]
Matsuyama, E., Lee, Y., Takahashi, N. and Tsai, D.Y. (2019）A Wavelet Coefficient-Based Convolutional Neural Network for Histological Classification of Lung Cancer in CT Images. Japanese Journal of Imaging and Information Sciences in Medicine (In Japanese), 36, 64-71.
[20]
Zech, J.R., Badgeley, M.A., Liu, M., Costa, A.B., Titano, J. and Oermann, E.K. (2018) Variable Generalization Performance of a Deep Learning Model to Detect Pneumonia in Chest Radiographs: A Cross-Sectional Study. PLOS Medicine, 15, e1002683. https://doi.org/10.1371/journal.pmed.1002683
[21]
Ovadia, Y., Fertig, E., Ren, J., Nado. Z., Sculley, D., Nowozin, S., et al. (2019) Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift. 33rd International Conference on Neural Information Processing Systems, Vancouver, 8-14 December 2019, 13969-13980.
[22]
Guo, C., Pleiss, G., Sun Y. and Weinberger, K.Q. (2017）On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, Vol. 70, 1321-1330. https://proceedings.mlr.press/v70/guo17a.html
[23]
Gal, Y. and Ghahramani, Z. (2016) Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. Proceedings of the 33rd International Conference on Machine Learning, Vol. 48, 1050-1059.
He, K., Zhang, X., Ren, S. and Sun, J. (2015) Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, 27-30 June 2016, 770-778. https://doi.org/10.1109/CVPR.2016.90
[26]
Narayanan, B.N., De Silva, M.S., Hardie, R.C., Kueterman, N.K. and Ali, R. (2019) Understanding Deep Neural Network Predictions for Medical Imaging Applications.
[27]
van der Maaten, L.J.P. and Hinton, G.E. (2008) Visualizing High-Dimensional Data Using t-SNE. Journal of Machine Learning Research, 9, 2579-2605.
[28]
Shan, B. and Fang, Y. (2020) A Cross Entropy Based Deep Neural Network Model for Road Extraction from Satellite Images. Entropy, 22, Article No. 535. https://doi.org/10.3390/e22050535
[29]
Kurian, N.C., Meshram, P.S., Patil, A., Patel S. and Sethi, A. (2021) Sample Specific Generalized Cross Entropy for Robust Histology Image Classification. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Nice, 13-16 April 2021, 1934-1938. https://doi.org/10.1109/ISBI48211.2021.9434169
[30]
Mannor, S., Peleg, D. and Rubinstein, R. (2005) The Cross Entropy Method for Classification. Proceedings of the 22nd International Conference on Machine Learning, Bonn, 7-11 August 2005, 561-568. https://doi.org/10.1145/1102351.1102422
[31]
Brownlee, J. (2020) A Gentle Introduction to Cross-Entropy for Machine Learning. https://machinelearningmastery.com/cross-entropy-for-machine-learning/
[32]
Mao, A., Mohri, M. and Zhong, Y. (2023) Cross-Entropy Loss Functions: Theoretical Analysis and Applications. Proceedings of the 40th International Conference on Machine Learning, Honolulu, Vol. 202, 23803-23828. https://proceedings.mlr.press/v202/mao23b/mao23b.pdf
[33]
Nova (2023) A Comprehensive Guide to Cross Entropy in Machine Learning. https://aitechtrend.com/a-comprehensive-guide-to-cross-entropy-in-machine-learning/
[34]
Sheikh, I. (2023) Understanding Cross-Entropy Loss and Its Role in Classification Problems. https://medium.com/@l228104/understanding-cross-entropy-loss-and-its-role-in-classification-problems-d2550f2caad5
