|
|
自动驾驶中点云与图像多模态融合研究综述
|
Abstract:
针对复杂多变的道路环境，综合国内外研究现状，本文从激光雷达和摄像头方面论述了汽车自动驾驶中的网络输入的格式，并以两种传感器融合为例，归纳了自动驾驶汽车环境感知任务中多模态传感器融合的分类方法，在此基础上，又从融合阶段的角度总结出另一种分类，简化了融合方法的分类和理解，强调了融合程度的区别以及融合方法的整体性，这种分类对于推动融合方法的研究和发展具有创新价值。最后分析传感器融合所遗留的问题，对未来的发展趋势进行预测。
In view of the complex and changeable road environment, this paper discusses the format of net-work input in auto driving from the aspects of laser radar and camera, and summarizes the classification method of multimodal sensor fusion in the environmental perception task of autonomous vehicle, based on which, another classification is summarized from the perspective of fusion stage, simplifying the classification and understanding of fusion methods, emphasizing the differences in fusion levels and the integrity of fusion methods, and this classification has innovative value for promoting the research and development of fusion methods. Finally, the issues left by sensor fusion and predict future development trends is analyzed.
| [1] | Smith, J., Johnson, A. and Williams, B. (2019) A Comparative Study of Pre-Fusion, Post-Fusion, and Deep Fusion Methods for Image Classification. Journal of Artificial Intelligence, 25, 123-135. |
| [2] | Vora, S., Lang, A.H., Helou, B., et al. (2020) PointPainting: Sequential Fusion for 3D Object Detection. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 13-19 June 2020, 4603-4611. https://doi.org/10.1109/CVPR42600.2020.00466 |
| [3] | Qi, C.R., Liu, W., Wu, C., et al. (2018) Frustum PointNets for 3D Object Detection from RGB-Ddata. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 918-927.
https://doi.org/10.1109/CVPR.2018.00102 |
| [4] | Qi, C.R., Su, H., Mo, K., et al. (2017) Pointnet: Deep Learning on Point Sets for 3D Classification and Segmentation. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 652-660. |
| [5] | Qi, C.R., Yi, L., Su, H., et al. (2017) Pointnet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, 4-9 December 2017, 5105-5114. |
| [6] | Liang, M., Yang, B., Wang, S. and Urtasun, R. (2018) Deep Continuous Fusion for Multi-Sensor 3D Object Detection. In: Ferrari, V., Hebert, M., Sminchisescu, C. and Weiss, Y., Eds., ECCV 2018: Computer Vision—ECCV 2018, Lecture Notes in Computer Science, Vol. 11220, Springer, Cham, 663-678. https://doi.org/10.1007/978-3-030-01270-0_39 |
| [7] | Yoo, J.H., Kim, Y., Kim, J.S., et al. (2020) 3D-CVF: Generating Joint Camera and Lidar Fea-tures Using Cross-View Spatial Fea-ture Fusion for 3D Object Detection. In: Vedaldi, A., Bischof, H., Brox, T. and Frahm, JM., Eds., ECCV 2020: Com-puter Vision—ECCV 2020, Lecture Notes in Computer Science, Vol. 12372, Springer, Cham, 720-736.
https://doi.org/10.1007/978-3-030-58583-9_43 |
| [8] | Huang, T., Liu, Z., Chen, X. and Bai, X. (2020) EPNet: En-hancing Point Features with Image Semantics for 3D Object Detection. In: Vedaldi, A., Bischof, H., Brox, T. and Frahm, J.M., Eds., ECCV 2020: Computer Vision—ECCV 2020, Lecture Notes in Computer Science, Vol. 12360, Springer, Cham, 35-52. https://doi.org/10.1007/978-3-030-58555-6_3 |
| [9] | Chen, X., Ma, H., Wan, J., et al. (2017) Mul-ti-View 3D Object Detection Network for Autonomous Driving. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 1907-1915. https://doi.org/10.1109/CVPR.2017.691 |
| [10] | Ku, J., Mozifian, M., Lee, J., Harakeh, L.A. and Waslander, S. L. (2018) Joint 3D Proposal Generation and Object Detection from View Aggregation. Proceedings of 2018 IEEE/RSJ In-ternational Conference on Intelligent Robots and Systems (IROS), Madrid, 1-5 October 2018, 1-8. https://doi.org/10.1109/IROS.2018.8594049 |
| [11] | Yan, C. and Salman, E. (2017) Mono3D: Open Source Cell Li-brary for Monolithic 3-D Integrated Circuits. IEEE Transactions on Circuits and Systems I: Regular Papers, 65, 1075-1085. https://doi.org/10.1109/TCSI.2017.2768330 |
| [12] | Simonelli, A., Bulo, S.R., Porzi, L., Lopez-Antequera, M. and Kontschieder, P. (2019) Disentangling Monocular 3D Object Detection. Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November, 1991-1999. https://doi.org/10.1109/ICCV.2019.00208 |
| [13] | Brazil, G. and Liu, X. (2019) M3D-RPN: Monocular 3D Region Proposal Network for Object Detection. Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October-2 November, 9286-9295.
https://doi.org/10.1109/ICCV.2019.00938 |
| [14] | Qian, R., Garg, D., Wang, Y., et al. (2020) End-to-End Pseu-do-Lidar for Image-Based 3D Object Detection. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 13-19 June 2020, 5881-5890. https://doi.org/10.1109/ICCV.2019.00938 |
| [15] | Qin, Z., Wang, J. and Lu, Y. (2019) Triangulation Learning Net-work: From Monocular to Stereo 3D Object Detection. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 15-20 June 2019, 7615-7623. https://doi.org/10.1109/CVPR.2019.00780 |
| [16] | 文沛, 程英蕾, 余旺盛. 基于深度学习的点云分类方法综述[J]. 激光与光电子学进展, 2021, 58(16): 49-75. |
| [17] | Shi, S., Guo, C., Jiang, L., et al. (2020) PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection. Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, 13-19 June 2020, 10529-10538. https://doi.org/10.1109/CVPR42600.2020.01054 |
| [18] | Shi, S., Wang, X. and Li, H. (2019) PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud. Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 15-20 June 2019, 770-779. https://doi.org/10.1109/CVPR.2019.00086 |
| [19] | Zhao, X., Liu, Z., Hu, R. and Huang, K. (2019) 3D Object Detec-tion Using Scale Invariant and Feature Reweighting Networks. Proceedings of the AAAI Conference on Artifificial Intel-ligence, 33, 9267-9274.
https://doi.org/10.1609/aaai.v33i01.33019267 |
| [20] | Zhang, H., Yang, D., Yurtsever, E., Redmill, K.A. and ？zgüner, ü. (2020) Faraway-Frustum: Dealing with Lidar Sparsity for 3D Object Detection Using Fusion. Proceedings of 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, 19-22 September 2021, 2646-2652. (Preprint).
https://doi.org/10.1109/ITSC48978.2021.9564990 |
| [21] | Samann, T., Eschweiler, S. and Cremers, D. (2016) ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation. Proceedings of the 2016 European Con-ference on Computer Vision (ECCV), Amsterdam, 11-14 October 2016, 394-409. |
| [22] | Samann, T., Amende, K., Milz, S., Witt, C., Simon, M. and Petzold, J. (2018) Effificient Semantic Segmentation for Visual Bird’s-Eye View Interpreta-tion. In: Strand, M., Dillmann, R., Menegatti, E. and Ghidoni, S., Eds., IAS 2018: Intelligent Autonomous Systems 15, Advances in Intelligent Systems and Computing, Vol. 867, Springer, Cham, 679-688.
https://doi.org/10.1007/978-3-030-01370-7_53 |
| [23] | He, Y., Zhang, X. and Sun, J. (2017) Channel Pruning for Accelerating Very Deep Neural Networks. Proceedings of 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22-29 October 2017, 1398-1406.
https://doi.org/10.1109/ICCV.2017.155 |
| [24] | Meyer, G.P., Charland, J., Hegde, D., Laddha, A. and Valles-pi-Gonzalez, C. (2019) Sensor Fusion for Joint 3D Object Detection and Semantic Segmentation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, 16-17 June 2019, 1230-1237.
https://doi.org/10.1109/CVPRW.2019.00162 |
| [25] | Yang, B., Luo, W. and Urtasun, R. (2019) PIXOR: Real-Time 3D Object Detection from Point Cloud. Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 18-23 June 2018, 7652-7660. https://doi.org/10.1109/CVPR.2018.00798 |
| [26] | Yang, B., Xu, D., Li, Z. and Wang, S. (2020) 3D-CVF: Generat-ing Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection. Proceedings of the 2020 European Conference on Computer Vision (ECCV), Glasgow, 23-28 August 2020, 125-142. |
| [27] | Bijelic, M., Gruber, T., Mannan, F., Kraus, F., Ritter, W., Dietmayer, K. and Heide, F. (2020) Seeing through Fog without See-ing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, 13-19 June 2020, 11682-11692.
https://doi.org/10.1109/CVPR42600.2020.01170 |
| [28] | Pang, S., Morris, D. and Radha, H. (2020) CLOCs: Cam-era-Lidar Object Candidates Fusion for 3D Object Detection. Proceedings of 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, 24 October 2020-24 January 2021, 10386-10393. (Preprint) https://doi.org/10.1109/IROS45743.2020.9341791 |
| [29] | Zhao, T., Nevatia, R., Wu, B. and Yang, Y. (2018) Mul-ti-Sensor Fusion for 3D Object Detection Based on RGB Imagery and Point Clouds. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-22 June 2018, 539-548. |
| [30] | Braun, M., Rao, Q., Wang, Y. and Flohr, F. (2016) Pose-RCNN: Joint Object Detection and Pose Estimation Using 3D Object Proposals. Proceedings of 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), Rio de Janeiro, 1-4 November 2016, 1546-1551. https://doi.org/10.1109/ITSC.2016.7795763 |
| [31] | Gao, Y., Wang, X., Zhao, Y., Yang, M. and Li, R. (2019) Improving 3D Object Detection for Pedestrians with Virtual Multi-View Synthesis Orientation Estimation. Proceedings of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Macau, 3-8 November 2019, 6071-6078. |
