|
|
一种基于多层卷积稀疏网络的红外与可见光图像融合方法
|
Abstract:
红外图像和可见光图像融合广泛应用于夜视、监视、军事等领域。融合任务的重点在于将可见光和红外光图像中的互补信息整合起来并消除多余信息。此外，大多数融合任务是在低光环境下进行的，如何保持融合结果的照明信息值得研究。为了解决存在的问题，首先，我们设计了一个多级特征模块来融合多源信息。与传统网络的并行层融合策略不同，我们提出了一种并行层和深度层相结合的融合策略。其次，我们在特征提取网络中增加了注意力计算，以提高特征提取网络的性能。第三，为了使融合图像具有良好的照明信息，我们设计了区域照明保留模块，提高了低光环境下融合算法的性能。大量实验证明了所提出的方法具有出色的性能，并且在低光环境下表现更好。此外，所提出的算法在多模式物体检测方面也显示出巨大潜力。
Infrared image and visible light image fusion is widely used in night vision, surveillance, military and other fields. The focus of the fusion task is to integrate complementary information in visible and infrared light images and eliminate redundant information. In addition, most of the fusion tasks are performed in the harsh environment of low light, and it is worth studying how to maintain the lighting information of the fusion results. In order to solve the problems existing, firstly, we de-sign a multi-level feature module to fusion multi-source information. Different from the parallel layer fusion strategy of the traditional network, we proposed a fusion strategy that combined par-allel layers and depth layers. Secondly, we add attention computing to the feature extraction net-work to improve the performance of the feature extraction network. Thirdly, in order to make the fusion image have good illumination information, we design the area illumination retention module, improving the performance of the fusion algorithm in low-light environments. A large number of experiments show that the proposed method has excellent performance and will perform better in low-light environments. In addition, the proposed algorithm also shows great potential in multi modal-object detection.
| [1] | Zhang, J., Lei, W., Li, S., Li, Z. and Li, X. (2023) Infrared and Visible Image Fusion Withentropy-Based Adaptive Fu-sion Module and Mask-Guided Convolutional Neural Network. Infrared Physics & Technology, 131, Article ID: 104629. https://doi.org/10.1016/j.infrared.2023.104629 |
| [2] | Ma, J., Ma, Y. and Li, C. (2019) Infrared and Visible Image Fusion Methods and Applications: A Survey. Information Fusion, 45, 153-178. https://doi.org/10.1016/j.inffus.2018.02.004 |
| [3] | Ma, J. and Zhou, Y. (2020) Infrared and Visible Image Fusion via Gradientlet Filter. Computer Vision and Image Understanding, 197-198, Article ID: 103016. https://doi.org/10.1016/j.cviu.2020.103016 |
| [4] | Xing, C., Wang, Z., Ouyang, Q., Dong, C. and Duan, C. (2019) Image Fusion Method Based on Spatially Masked Convolutional Sparse Represenitation. Image and Vision Computing, 90, Article ID: 103806.
https://doi.org/10.1016/j.imavis.2019.08.010 |
| [5] | Bavirisetti, D.P. and Dhuli, R. (2016) Two-Scale Image Fusion of Visible and Infrared Images Using Saliency Detection. Infrared Physics & Technology, 76, 52-64. https://doi.org/10.1016/j.infrared.2016.01.009 |
| [6] | Li, H., Cen, Y., Liu, Y., Chen, X. and Yu, Z. (2021) Different Input Resolutions and Arbitrary Output Resolution: A Meta Learning-Based Deep Framework for Infrared and Visible Image Fusion. IEEE Transactions on Image Processing, 30, 4070-4083. https://doi.org/10.1109/TIP.2021.3069339 |
| [7] | Jian, L., Yang, X., Liu, Z., Jeon, G. and Chisholm, D. (2020) SEDRFuse: A Symmetric Encoder-Decoder with Residual Block Network for Infrared and Visible Image Fusion. IEEE Transactions on Instrumentation and Measurement, 70, 1-15. https://doi.org/10.1109/TIM.2020.3022438 |
| [8] | Liu, R., Liu, J., Jiang, Z., Fan, X. and Luo, Z. (2020) A Bilevel Integrated Model with Data-Driven Layer Ensemble for Mul-ti-Modality Image Fusion. IEEE Transactions on Image Processing, 30, 1261-1274.
https://doi.org/10.1109/TIP.2020.3043125 |
| [9] | Yang, Y., Liu, J., Huang, S., Wan, W. and Guan, J. (2021) Infra-red and Visible Image Fusion via Texture Conditional Generative Adversarial Network. IEEE Transactions on Circuits and Systems for Video Technology, 31, 4771-4783.
https://doi.org/10.1109/TCSVT.2021.3054584 |
| [10] | Zhou, H., Wu, W., Zhang, Y., Ma, J. and Ling, H. (2023) Se-mantic-Supervised Infrared and Visible Image Fusion via a Dual-Discriminator Generative Adversarial Network. IEEE Transactions on Multimedia, 25, 635-648.
https://doi.org/10.1109/TMM.2021.3129609 |
| [11] | Zhang, H. and Ma, J. (2021) SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion. International Journal of Computer Vision, 129, 2761-2785. https://doi.org/10.1007/s11263-021-01501-8 |
| [12] | Ma, C. (2019) FusionGAN: A Generative Adver-sarial Network for Infrared and Visible Image Fusion. Information Fusion, 48, 11-26. https://doi.org/10.1016/j.inffus.2018.09.004 |
| [13] | Li, S., Kang, X. and Hu, J. (2013) Image Fusion with Guided Filtering. IEEE Transactions on Image Processing, 22, 2864-2875. https://doi.org/10.1109/TIP.2013.2244222 |
| [14] | Hui, L., Wu, X.J. and Kittler, J. (2018) Infrared and Vrisible Image Fusion Using a Deep Learning Framework. 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, 20-24 August 2018, 2705-2710. |
| [15] | Hui, L. and Wu, X.J. (2018) DenseFuse: A Fusion Approach to Infrared and Visible Images. IEEE Transactions on Image Processing, 28, 2614-2623. https://doi.org/10.1109/TIP.2018.2887342 |
| [16] | Ma, J., Tang, L., Xu, M., Zhang, H. and Xiao, G. (2021) STDFusionNet: An Infrared and Visible Image Fusion Network Based on Salient Target Detection. IEEE Transactions on Instrumentation and Measurement, 70, 1-13.
https://doi.org/10.1109/TIM.2021.3075747 |
| [17] | Tang, L., Yuan, J. and Ma, J. (2022) Image Fusion in the Loop of High-Level Vision Tasks: A Semantic-Aware Real-Time Infrared and Visible Image Fusion Network. Information Fu-sion, 82, 28-42.
https://doi.org/10.1016/j.inffus.2021.12.004 |
| [18] | Li, X. (2021) RFN-Nest: An End-to-End Residual Fusion Net-work for Infrared and Visible Images. Information Fusion, 73, 72-86. https://doi.org/10.1016/j.inffus.2021.02.023 |
| [19] | Tang, L., Yuan, J., Zhang, H., Jiang, X. and Ma, J. (2022) PIA-Fusion: A Progressive Infrared and Visible Image Fusion Network Based on Illumination Aware. Information Fusion, 83-84, 79-92.
https://doi.org/10.1016/j.inffus.2022.03.007 |
| [20] | Xie, H., Zhang, Y., Qiu, J., Zhai, X., Liu, X., Yang, Y., Zhao, S., Luo, Y. and Zhong, J. (2023) Semantics Lead All: Towards Unified Image Registration and Fusion from a Semantic Perspective. Information Fusion, 98, Article ID: 101835. https://www.sciencedirect.com/science/article/pii/S1566253523001513 https://doi.org/10.1016/j.inffus.2023.101835 |
| [21] | Tang, L., Liu, G., Xiao, G., Bavirisetti, D.P. and Zhang, X. (2022) Infrared and Visible Image Fusion Based on Guided Hybrid Model and Generative Adversarial Network. Infra-red Physics & Technology, 120, Article ID: 103914.
https://doi.org/10.1016/j.infrared.2021.103914 |
| [22] | Liu, X., Wang, R., Huo, H., Yang, X. and Li, J. (2023) An Attention-Guided and Wavelet-Constrained Generative Adversarial Network for Infrared and Visible Image Fusion. In-frared Physics & Technology, 129, Article ID: 104570.
https://doi.org/10.1016/j.infrared.2023.104570 |
| [23] | Xu, H., Liang, P., Yu, W., Jiang, J. and Ma, J. (2019) Learning a Generative Model for Fusing Infrared and Visible Images via Conditional Generative Adversarial Network with Dual Discriminators. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, In-ternational Joint Conferences on Artificial Intelligence Organization, Macao, 10-16 August 2019, 3954-3960. https://doi.org/10.24963/ijcai.2019/549 |
| [24] | Ma, J., Xu, H., Jiang, J., Mei, X. and Zhang, X.P. (2020) DDcGAN: A Dual-Discriminator Conditional Generative Adversarial Network for Multi-Resolution Image Fusion. IEEE Transac-tions on Image Processing, 29, 4980-4995.
https://doi.org/10.1109/TIP.2020.2977573 |
| [25] | Zhang, H., Yuan, J., Tian, X. and Ma, J. (2021) GAN-FM: Infra-red and Visible Image Fusion Using GAN with Full-Scale Skipconnection and Dual Markovian Discriminators. IEEE Transactions on Computational Imaging, 7, 1134-1147. https://doi.org/10.1109/TCI.2021.3119954 |
| [26] | Li, J., Huo, H.T., Li, C., Wang, R. and Feng, Q. (2020) AttentionFGAN: Infrared and Visible Image Fusion Using Atten-tion-Based Generative Aidversarial Networks. IEEE Transactions on Multimedia, 23, 1383-1396.
https://doi.org/10.1109/TMM.2020.2997127 |
| [27] | Rao, Y., Wu, D., Han, M., Wang, T., Yang, Y., Lei, T., Zhou, C., Bai, H. and Xing, L. (2023) AT-GAN: A Generative Adversarial Network with Attention and Transition for Infrared and Visible Image Fusion. Information Fusion, 92, 336-349. https://www.sciencedirect.com/science/article/pii/S156625352200255X https://doi.org/10.1016/j.inffus.2022.12.007 |
| [28] | Jie, H., Li, S., Gang, S. and Albanie, S. (2017) Squeeze-and-Excitation Networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 7132-7141. |
| [29] | Toet, A. (2022) TNOimage Fusion Dataset.
https://figshare.com/articles/dataset/TNO_Image_Fusion_Dataset/1008029 |
| [30] | Xu, H., Ma, J., Jiang, J., Guo, X. and Ling, H. (2022) U2fusion: A Unified Unsupervised Image Fusion Network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 502-518.
https://doi.org/10.1109/TPAMI.2020.3012548 |
