This paper studies the effect of breaking single-class building data into multi-class building data for semantic segmentation under end-to-end architecture such as UNet, UNet++, DeepLabV3, and DeepLabv3+. Although, the already existing semantic segmentation methods for building detection work on the imagery of developed world, where the buildings are highly structured and there is a clearly distinguishable space present between the building instances, the same methods do not work as effectively on the developing world where there is often no clear differentiable spaces between instances of building thus reducing the number of detected instances. Hence as a noble approach, we have added building contours as new class along with building segmentation data, and detected the building contours and the inner building regions, hence giving the precise number of buildings existing in the input imagery especially in the convoluted areas where the boundary between the buildings are often hard to determine even for human eyes. Breaking down the building data into multi-class data increased the building detection precision and recall. This is useful in building detection where building instances are convoluted and are difficult for bare instance segmentation to detect all the instances.
References
[1]
Barbedo, J.G.A., Koenigkan, L.V., Santos, P.M. and Ribeiro, A.R.B. (2020) Counting Cattle in UAV Images—Dealing with Clustered Animals and Animal/Background Contrast Changes. Sensors, 20, Article No. 2126. https://doi.org/10.3390/s20072126
[2]
Hamdi, Z.M., Brandmeier, M. and Straub, C. (2019) Forest Damage Assessment Using Deep Learning on High Resolution Remote Sensing Data. Remote Sensing, 11, Article No. 1976. https://doi.org/10.3390/rs11171976
[3]
Giang, T.L., Dang, K.B., Le, Q.T., Nguyen, V.G., Tong, S.S. and Pham, V.-M. (2020) U-Net Convolutional Networks for Mining Land Cover Classification Based on High-Resolution UAV Imagery. IEEE Access, 8, 186257-186273. https://doi.org/10.1109/ACCESS.2020.3030112
[4]
Gebrehiwot, A., Hashemi-Beni, L., Thompson, G., Kordjamshidi, P. and Langan, T. (2019) Deep Convolutional Neural Network for Flood Extent Mapping Using Unmanned Aerial Vehicles Data. Sensors, 19, Article No. 1486. https://doi.org/10.3390/s19071486
[5]
Boonpook, W., Tan, Y. and Xu, B. (2021) Deep Learning-Based Multi-Feature Semantic Segmentation in Building Extraction from Images of UAV Photogrammetry. International Journal of Remote Sensing, 42, 1-19. https://doi.org/10.1080/01431161.2020.1788742
[6]
Orfanidis, G., Ioannidis, K., Avgerinakis, K., Vrochidis, S. and Kompatsiaris, I. (2018) A Deep Neural Network for Oil Spill Semantic Segmentation in Sar Images. 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, Athens, 7-10 October 2018, 3773-3777. https://doi.org/10.1109/ICIP.2018.8451113
[7]
Kumar, L., Sinha, P. and Taylor, S. (2014) Improving Image Classification in a Complex Wetland Ecosystem through Image Fusion Techniques. Journal of Applied Remote Sensing, 8, Article ID: 083616. https://doi.org/10.1117/1.JRS.8.083616
[8]
Wu, M., Zhang, C., Liu, J., Zhou, L. and Li, X. (2019) Towards Accurate High Resolution Satellite Image Semantic Segmentation. IEEE Access, 7, 55609-55619. https://doi.org/10.1109/ACCESS.2019.2913442
[9]
Mesner, N. and Ostir, K. (2014) Investigating the Impact of Spatial and Spectral Resolution of Satellite Images on Segmentation Quality. Journal of Applied Remote Sensing, 8, Article ID: 083696. https://doi.org/10.1117/1.JRS.8.083696
[10]
Liu, J., Li, P. and Wang, X. (2015) A New Segmentation Method for Very High Resolution Imagery Using Spectral and Morphological Information. ISPRS Journal of Photogrammetry and Remote Sensing, 101, 145-162. https://doi.org/10.1016/j.isprsjprs.2014.11.009
[11]
Osco, L.P., Junior, J.M., Marques Ramos, A.P., De Castro Jorge, L.A., Fatholahi, S.N., De Andrade Silva, J., Matsubara, E.T., Pistori, H., Gonçalves, W.N. and Li, J. (2021) A Review on Deep Learning in UAV Remote Sensing. International Journal of Applied Earth Observation and Geoinformation, 102, Article ID: 102456. https://doi.org/10.1016/j.jag.2021.102456
[12]
Diakogiannis, F., Waldner, F., Caccetta, P. and Wu, C. (2020) ResUNET-A: A Deep Learning Framework for Semantic Segmentation of Remotely Sensed Data. ISPRS Journal of Photogrammetry and Remote Sensing, 162, 94-114. https://doi.org/10.1016/j.isprsjprs.2020.01.013
[13]
Lateef, F. and Ruichek, Y. (2019) Survey on Semantic Segmentation Using Deep Learning Techniques. Neurocomputing, 338, 321-348. https://doi.org/10.1016/j.neucom.2019.02.003
[14]
Zhao, K., Kang, J., Jung, J. and Sohn, G. (2018) Building Extraction from Satellite Images Using Mask R-CNN with Building Boundary Regularization. 2018 IEEE/ CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, 18-22 June 2018, 242-2424. https://doi.org/10.1109/CVPRW.2018.00045
[15]
Li, W., He, C., Fang, J., Zheng, J., Fu, H. and Yu, L. (2019) Semantic Segmentation-Based Building Footprint Extraction Using Very High-Resolution Satellite Images and Multi-Source GIS Data. Remote Sensing, 11, Article No. 403. https://doi.org/10.3390/rs11040403
[16]
Xu, Y., Wu, L., Xie, Z. and Chen, Z. (2018) Building Extraction in Very High Resolution Remote Sensing Imagery Using Deep Learning and Guided Filters. Remote Sensing, 10, Article No. 144. https://doi.org/10.3390/rs10010144
[17]
Audebert, N., Le Saux, B. and Lefèvre, S. (2017) Segment-before-Detect: Vehicle Detection and Classification through Semantic Segmentation of Aerial Images. Remote Sensing, 9, Article No. 368. https://doi.org/10.3390/rs9040368
[18]
Wu, G., Shao, X., Guo, Z., Chen, Q., Yuan, W., Shi, X., Xu, Y. and Shibasaki, R. (2018) Automatic Building Segmentation of Aerial Imagery Using Multi-Constraint Fully Convolutional Networks. Remote Sensing, 10, Article No. 407. https://doi.org/10.3390/rs10030407
[19]
OpenCV (2023) Eroding and Dilating. https://docs.opencv.org/3.4/db/df6/tutorial_erosion_dilatation.html
[20]
Open Cities AI Challenge Dataset. Version 1.0, Radiant MLHub. https://mlhub.earth/10.34911/rdnt.f94cxb
[21]
Maggiori, E., Tarabalka, Y., Charpiat, G. and Alliez, P. (2017) Can Semantic Labeling Methods Generalize to Any City? The Inria Aerial Image Labeling Benchmark. 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), Fort Worth, 23-28 July 2017, 3226-3229. https://doi.org/10.1109/IGARSS.2017.8127684
[22]
Dixon, B. and Candade, N. (2008) Multispectral Landuse Classification Using Neural Networks and Support Vector Machines: One or the Other, or Both? International Journal of Remote Sensing, 29, 1185-1206. https://doi.org/10.1080/01431160701294661
[23]
Singh, D., Maurya, R., Shukla, A., Sharma, M. and Gupta, P. (2012) Building Extraction from Very High Resolution Multispectral Images Using NDVI Based Segmentation and Morphological Operators. 2012 Students Conference on Engineering and Systems, Allahabad, 16-18 March 2012, 1-5. https://doi.org/10.1109/SCES.2012.6199034
[24]
Huang, X. and Zhang, L. (2012) Morphological Building/Shadow Index for Building Extraction from High-Resolution Imagery Over Urban Areas. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 5, 161-172. https://doi.org/10.1109/JSTARS.2011.2168195
[25]
You, Y., et al. (2018) Building Detection from VHR Remote Sensing Imagery Based on the Morphological Building Index. Remote Sensing, 10, Article No. 1287. https://doi.org/10.3390/rs10081287
[26]
Lefevre, S., Weber, J. and Sheeren, D. (2007) Automatic Building Extraction in VHR Images Using Advanced Morphological Operators. 2007 Urban Remote Sensing Joint Event, Paris, 11-13 April 2007, 1-5. https://doi.org/10.1109/URS.2007.371825
[27]
Kulathunga, G.P. and Afanasyev, I. (2018) Deep Learning Approach for Building Detection in Satellite Multispectral Imagery. https://www.researchgate.net/publication/328899673_Deep_Learning_Approach_for_Building_Detection_in_Satellite_Multispectral_Imagery
[28]
Zhang, P., Du, P., Lin, C., Wang, X., Li, E., Xue, Z. and Bai, X. (2020) A Hybrid Attention-Aware Fusion Network (HAFNet) for Building Extraction from High-Resolution Imagery and LiDAR Data. Remote Sensing, 12, Article No. 3764. https://doi.org/10.3390/rs12223764
[29]
Krizhevsky, A., Sutskever, I. and Hinton, G. (2012) ImageNet Classification with Deep Convolutional Neural Networks. In: Pereira, F., Burges, C.J., Bottou, L. and Weinberger, K.Q., Eds., Advances in Neural Information Processing Systems 25, Curran Associates, Inc., Red Hook, 1097-1105.
[30]
He, K., Gkioxari, G., Dollár, P. and Girshick, R. (2017) Mask R-CNN. 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22-29 October 2017, 2980-2988. https://doi.org/10.1109/ICCV.2017.322
[31]
Ronneberger, O., Fischer, P. and Brox, T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab, N., Hornegger, J., Wells, W. and Frangi, A., Eds., Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351, Springer, Cham, 234-241. https://doi.org/10.1007/978-3-319-24574-4_28
[32]
Liu, X., Song, L., Liu, S. and Zhang, Y. (2021) A Review of Deep-Learning-Based Medical Image Segmentation Methods. Sustainability, 13, Article No. 1224. https://doi.org/10.3390/su13031224
[33]
Yeung, H.W.F., Zhou, M., Chung, Y.Y., Moule, G., Thompson, W., Ouyang, W., Cai, W. and Bennamoun, M. (2022) Deep-Learning-Based Solution for Data Deficient Satellite Image Segmentation. Expert Systems with Applications, 191, Article ID: 116210. https://doi.org/10.1016/j.eswa.2021.116210
[34]
Kebaili, A., Lapuyade-Lahorgue, J. and Ruan, S. (2023) Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review. Journal of Imaging, 9, Article No. 81. https://doi.org/10.3390/jimaging9040081
[35]
Weir, N., Lindenbaum, D., Bastidas, A., Etten, A., Kumar, V., Mcpherson, S., Shermeyer, J. and Tang, H. (2019) Spacenet MVOI: A Multi-View Overhead Imagery Dataset. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, South Korea, 27 October 2019 - 02 November 2019, 992-1001.
[36]
Wagner, F.H., Silva, R., Tarabalka, Y., Segantine, T., Thomé, R. and Hirye, M. (2020) U-Net-Id, an Instance Segmentation Model for Building Extraction from Satellite Images—Case Study in the Joanópolis City, Brazil. Remote Sensing, 12, Article No. 1544. https://doi.org/10.3390/rs12101544
[37]
Liao, C., Hu, H., Li, H., Ge, X., Chen, M., Li, C. and Zhu, Q. (2021) Joint Learning of Contour and Structure for Boundary-Preserved Building Extraction. Remote Sensing, 13, Article No. 1049. https://www.mdpi.com/2072-4292/13/6/1049 https://doi.org/10.3390/rs13061049
[38]
Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N. and Liang, J. (2018) UNet++: A Nested U-Net Architecture for Medical Image Segmentation. In: Stoyanov, D., et al., Eds., Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA ML-CDS 2018. Lecture Notes in Computer Science, Vol. 11045, Springer, Cham, 3-11. https://doi.org/10.1007/978-3-030-00889-5_1 https://link.springer.com/chapter/10.1007/978-3-030-00889-5_1
[39]
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A. (2017) DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. http://arxiv.org/abs/1606.00915
[40]
Chen, L.-C., Papandreou, G., Schroff, F. and Adam, H. (2017) Rethinking Atrous Convolution for Semantic Image Segmentation. ArXiv: 1706.05587.
[41]
Tan, M. and Le, Q. (2019) EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. Proceedings of the 36th International Conference on Machine Learning, Long Beach, 9-15 June 2019, 6105-6114. https://proceedings.mlr.press/v97/tan19a.html
[42]
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. and Li, F.-F. (2009) ImageNet: A Large-Scale Hierarchical Image Database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, 20-25 June 2009, 248-255. https://doi.org/10.1109/CVPR.2009.5206848
[43]
Sudre, C.H., Li,W., Vercauteren, T., Ourselin, S. and Jorge Cardoso, M. (2017) Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations. In: Cardoso, M., et al., Eds., Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA ML-CDS 2017. Lecture Notes in Computer Science, Vol. 10553, Springer, Cham, 240-248. https://doi.org/10.1007/978-3-319-67558-9_28
[44]
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I. and Savarese, S. (2019) Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 15-20 June 2019, 658-666. https://doi.org/10.1109/CVPR.2019.00075
[45]
Google for Developers (2023) Classification: Precision and Recall. https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall
[46]
Paszke, A., et al. (2019) Pytorch: An Imperative Style, High-Performance Deep Learning Library. 33rd Annual Conference on Neural Information Processing Systems, Vancouver, 8-14 December 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf
[47]
Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M. and Kalinin, A.A. (2020) Albumentations: Fast and Flexible Image Augmentations. Information, 11, Article No. 125. https://doi.org/10.3390/info11020125
[48]
Perez, L. and Wang, J. (2017) The Effectiveness of Data Augmentation in Image Classification Using Deep Learning. ArXiv: 1712.04621.
[49]
Lorensen, W.E. and Cline, H.E. (1987) Marching Cubes: A High Resolution 3d Surface Construction Algorithm. ACM SIGGRAPH Computer Graphics, 21, 163-169. https://doi.org/10.1145/37402.37422
[50]
van der Walt, S., Schönberger, J.L., Nunez-Iglesias, J., Boulogne, F., Warner, J.D., Yager, N., Gouillart, E. and Yu, T. (2014) Scikit-Image: Image Processing in Python. PeerJ, 2, e453. https://doi.org/10.7717/peerj.453
[51]
Ramer, U. (1972) An Iterative Procedure for the Polygonal Approximation of Plane Curves. Computer Graphics and Image Processing, 1, 244-256. https://doi.org/10.1016/S0146-664X(72)80017-0
