Critical to the safe, efficient, and reliable operation of an autonomous maritime vessel is its ability to perceive the external environment through onboard sensors. For this research, data was collected from a LiDAR sensor installed on a 16-foot catamaran unmanned vessel. This sensor generated point clouds of the surrounding maritime environment, which were then labeled by hand for training a machine learning (ML) model to perform a semantic segmentation task on LiDAR scans. In particular, the developed semantic segmentation classifies each point-cloud point as belonging to a certain buoy type. This paper describes the developed Unity Game Engine (Unity) simulation to emulate the maritime environment perceived by LiDAR with the goal of generating large (automatically labeled) simulation datasets and improving the ML model performance since hand-labeled real-life LiDAR scan data may be scarce. The Unity simulation data combined with labeled real-life point cloud data was used for a PointNet-based neural network model, the architecture of which is presented in this paper. Fitting the PointNet-based model on the simulation data followed by fine-tuning the combined dataset allowed for accurate semantic segmentation of point clouds on the real-world data. The ML model performance on several combinations of simulation and real-life data is explored. The resulting Intersection over Union (IoU) metric scores are quite high, ranging between 0.78 and 0.89, when validated on simulation and real-life data. The confusion matrix-entry values indicate an accurate semantic segmentation of the buoy types.
References
[1]
Saglam, A. and Papelis, Y. (2023) Efficient Maritime Object Detection and Validation for Enhancing Safety of Uncrewed Marine Systems. Proceedings of the 25th International Conference on Harbor, Maritime and Multimodal Logistic Modeling & Simulation (HMS 2023), Greece, 18-20 September 2023, 7.
[2]
Unity (2022) Unity Real-Time Development Platform. https://www.unity.com/
[3]
Velodyne Lidar, I. (2019) VLP-16 User Manual. https://www.manualslib.com/manual/1407706/Velodyne-Vlp-16.html
[4]
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A. and Koltun, V. (2017) CARLA: An Open Urban Driving Simulator. Proceedings of the 1st Annual Conference on Robot Learning, California, 13-15 November 2017, 1-16.
[5]
Jaiswal, C., Penumatcha, H., Varma, S., AlHmoud, I.W., Islam, A.K. and Gokaraju, B. (2024) Enriching 3D Object Detection in Autonomous Driving for Emergency Scenarios: Leveraging Point Cloud Data with CARLA Simulator for Automated Annotation of Rare 3D Objects. SoutheastCon 2024, Atlanta, 15-24 March 2024, 1137-1143. https://doi.org/10.1109/southeastcon52093.2024.10500173
[6]
Haarnoja, T., Moran, B., Lever, G., Huang, S.H., Tirumala, D., Humplik, J., et al. (2024) Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning. Science Robotics, 9, 1-17. https://doi.org/10.1126/scirobotics.adi8022
[7]
Qi, C.R., Yi, L., Su, H. and Guibas, L.J. (2017) Pointnet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space.
[8]
Lai, X., Chen, Y., Lu, F., Liu, J. and Jia, J. (2023) Spherical Transformer for Lidar-Based 3D Recognition. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, 17-24 June 2023, 17545-17555. https://doi.org/10.1109/cvpr52729.2023.01683
[9]
Epic Games (2024) Unreal Engine. https://www.unrealengine.com
[10]
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., et al. (2019) Semantickitti: A Dataset for Semantic Scene Understanding of Lidar Sequences. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 9296-9306. https://doi.org/10.1109/iccv.2019.00939
[11]
Simon, M., Amende, K., Kraus, A., Honer, J., Samann, T., Kaulbersch, H., et al. (2019) Complexer-Yolo: Real-Time 3D Object Detection and Tracking on Semantic Point Clouds. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, 16-17 June 2019, 1190-1199. https://doi.org/10.1109/cvprw.2019.00158
[12]
Charles, R.Q., Su, H., Kaichun, M. and Guibas, L.J. (2017) Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, 21-26 July 2017, 77-85. https://doi.org/10.1109/cvpr.2017.16
[13]
Ronneberger, O., Fischer, P. and Brox, T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Lecture Notes in Computer Science, Springer, 234-241. https://doi.org/10.1007/978-3-319-24574-4_28
[14]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., et al. (2023) Attention Is All You Need. https://arxiv.org/abs/1706.03762
[15]
Redmon, J., Divvala, S.K., Girshick, R.B. and Farhadi, A. (2015) You Only Look Once: Unified, Real-Time Object Detection. http://arxiv.org/abs/1506.02640
[16]
Scarselli, F., Gori, M., Ah Chung, T., Hagenbuchner, M. and Monfardini, G. (2009) The Graph Neural Network Model. IEEE Transactions on Neural Networks, 20, 61-80. https://doi.org/10.1109/tnn.2008.2005605
[17]
Thomas, H., Qi, C.R., Deschaud, J., Marcotegui, B., Goulette, F. and Guibas, L. (2019) KPConv: Flexible and Deformable Convolution for Point Clouds. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November 2019, 6410-6419. https://doi.org/10.1109/iccv.2019.00651
NVIDIA (2004) GPU Gems: Chapter 1: Effective Water Simulation from Physical Models. https://developer.nvidia.com/gpugems/gpugems/part-i-natural-effects/chapter-1-effective-water-simulation-physical-models
[22]
Zhu, J., Park, T., Isola, P. and Efros, A.A. (2017) Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22-29 October 2017, 2242-2251. https://doi.org/10.1109/iccv.2017.244
