|
|
基于交叉型窗口自注意力机制的Transformer临近预报
|
Abstract:
临近预报旨在预测未来0~2小时的天气情况，这对于减轻强对流天气的影响并保障人们的日常活动至关重要。作为当前天气预报中最具挑战性的任务之一，临近预报需要同时具备高精度和高时效性。为了应对这一挑战，本文提出了一种基于Transformer的编码器–解码器临近预报模型结构。该模型使用基于交叉型窗口的自注意力机制，从而高效捕捉天气雷达回波的全局特征；同时模型结合卷积块和高效的多尺度注意力机制，实现对多尺度局部特征的有效提取。模型通过引入门控机制有效地融合全局与局部特征，从而进一步提升模型预报性能。提出的模型在公开的上海雷达回波数据集上的实验结果有效验证了它的有效性和实用性。
Nowcasting aims to predict weather condition within the next 0~2 hours, which is critical for mitigating the impacts of severe convective weather and ensuring the safety of daily activities. As one of the most challenging tasks in modern weather forecasting, nowcasting demands both high accuracy and timeliness. To alleviate this challenge, this paper proposes a Transformer based on encoder-decoder architecture for nowcasting. The proposed model employs a cross-shaped window self-attention mechanism to efficiently capture the global features of radar echo maps. Additionally, it uses convolutional blocks and a multi-scale attention mechanism to effectively extract multi-scale local features. By incorporating a gating mechanism, the model effectively fuses global and local features, further enhancing forecasting performance. Experimental results demonstrate that the proposed model achieves superior performance on the public Shanghai radar echo dataset.
| [1] | 俞小鼎, 郑永光. 中国当代强对流天气研究与业务进展[J]. 气象学报, 2020, 78(3): 391-418. |
| [2] | Kimura, R. (2002) Numerical Weather Prediction. Journal of Wind Engineering and Industrial Aerodynamics, 90, 1403-1414. https://doi.org/10.1016/s0167-6105(02)00261-1 |
| [3] | Li, P.W. and Lai, E.S.T. (2004) Applications of Radar‐Based Nowcasting Techniques for Mesoscale Weather Forecasting in Hong Kong. Meteorological Applications, 11, 253-264. https://doi.org/10.1017/s1350482704001331 |
| [4] | Sun, J., Xue, M., Wilson, J.W., Zawadzki, I., Ballard, S.P., Onvlee-Hooimeyer, J., et al. (2014) Use of NWP for Nowcasting Convective Precipitation: Recent Progress and Challenges. Bulletin of the American Meteorological Society, 95, 409-426. https://doi.org/10.1175/bams-d-11-00263.1 |
| [5] | Wilson, J.W., Crook, N.A., Mueller, C.K., Sun, J. and Dixon, M. (1998) Nowcasting Thunderstorms: A Status Report. Bulletin of the American Meteorological Society, 79, 2079-2099. https://doi.org/10.1175/1520-0477(1998)079<2079:ntasr>2.0.co;2 |
| [6] | Ayzel, G., Heistermann, M. and Winterrath, T. (2019) Optical Flow Models as an Open Benchmark for Radar-Based Precipitation Nowcasting (Rainymotion V0.1). Geoscientific Model Development, 12, 1387-1402. https://doi.org/10.5194/gmd-12-1387-2019 |
| [7] | Prudden, R., Adams, S., Kangin, D., Robinson, N., et al. (2020) A Review of Radar-Based Nowcasting of Precipitation and Applicable Machine Learning Techniques. arXiv: 2005.04988. https://doi.org/10.48550/arXiv.2005.04988 |
| [8] | LeCun, Y., Bengio, Y. and Hinton, G. (2015) Deep Learning. Nature, 521, 436-444. https://doi.org/10.1038/nature14539 |
| [9] | Shi, X., Chen, Z., Wang, H., Yeung, D.Y., et al. (Year). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Annual Conference on Neural Information Processing Systems (NIPS 2015), Montreal, 7-12 December 2015, 802-810. https://proceedings.neurips.cc/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html |
| [10] | Graves, A. (2012) Long Short-Term Memory. In: Graves, A., Ed., Supervised Sequence Labelling with Recurrent Neural Networks, Springer, 37-45. https://doi.org/10.1007/978-3-642-24797-2_4 |
| [11] | Shi, X., Gao, Z., Lausen, L., Wang, H., et al. (Year) Deep Learning for Precipitation Nowcasting: A Benchmark and a New Model. Annual Conference on Neural Information Processing Systems (NIPS 2017), California, 4-9 December 2017, 5618-5628. https://proceedings.neurips.cc/paper/2017/hash/a6db4ed04f1621a119799fd3d7545d3d-Abstract.html |
| [12] | Wang, Y., Long, M., Wang, J., Gao, Z., et al. (Year) PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs. Annual Conference on Neural Information Processing Systems (NIPS 2017), California, 4-9 December 2017, 880-889. https://proceedings.neurips.cc/paper/2017/hash/e5f6ad6ce374177eef023bf5d0c018b6-Abstract.html |
| [13] | Wang, Y., Wu, H., Zhang, J., Gao, Z., Wang, J., Yu, P.S., et al. (2023) PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45, 2208-2225. https://doi.org/10.1109/tpami.2022.3165153 |
| [14] | Ronneberger, O., Fischer, P. and Brox, T. (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: Navab, N., Hornegger, J., Wells, W. and Frangi, A., Eds., Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Springer, 234-241. https://doi.org/10.1007/978-3-319-24574-4_28 |
| [15] | Agrawal, S., Barrington, L., Bromberg, C., Burge, J., et al. (2019). Machine Learning for Precipitation Nowcasting from Radar Images. arXiv: 1912.12132. https://doi.org/10.48550/arXiv.1912.12132 |
| [16] | Trebing, K., Staǹczyk, T. and Mehrkanoon, S. (2021) SmaAt-UNet: Precipitation Nowcasting Using a Small Attention-Unet Architecture. Pattern Recognition Letters, 145, 178-186. https://doi.org/10.1016/j.patrec.2021.01.036 |
| [17] | Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N. and Polosukhin, I. (2017) Attention Is All You Need. Advances in Neural Information Processing Systems, 30, 6000-6010. |
| [18] | Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al. (2021). An Image Is Worth 16 x 16 Words: Trans-formers for Image Recognition at Scale. International Conference on Learning Representations (ICLR), Vienna, 4 May 2021. |
| [19] | Xie, E., Wang, W., Yu, Z., Anandkumar, A., et al. (Year) SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. Conference on Neural Information Processing Systems (NeurIPS 2021), 6-14 December 2021, 12077-12090. https://proceedings.neurips.cc/paper/2021/hash/64f1f27bf1b4ec22924fd0acb550c235-Abstract.html |
| [20] | Chitta, K., Prakash, A., Jaeger, B., Yu, Z., Renz, K. and Geiger, A. (2023) Transfuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45, 12878-12895. https://doi.org/10.1109/tpami.2022.3200245 |
| [21] | Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., et al. (2022) UNETR: Transformers for 3D Medical Image Segmentation. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, 4-8 January 2022, 1748-1758. https://doi.org/10.1109/wacv51458.2022.00181 |
| [22] | Bai, C., Sun, F., Zhang, J., Song, Y. and Chen, S. (2022) Rainformer: Features Extraction Balanced Network for Radar-Based Precipitation Nowcasting. IEEE Geoscience and Remote Sensing Letters, 19, 1-5. https://doi.org/10.1109/lgrs.2022.3162882 |
| [23] | Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., et al. (2021) Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, 10-17 October 2021, 9992-10002. https://doi.org/10.1109/iccv48922.2021.00986 |
| [24] | Li, D., Deng, K., Zhang, D., Liu, Y., Leng, H., Yin, F., et al. (2023) LPT-QPN: A Lightweight Physics-Informed Transformer for Quantitative Precipitation Nowcasting. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-19. https://doi.org/10.1109/tgrs.2023.3328945 |
| [25] | Chen, S., Shu, T., Zhao, H., Zhong, G. and Chen, X. (2023) TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Autoregression. IEEE Transactions on Geoscience and Remote Sensing, 61, 1-14. https://doi.org/10.1109/tgrs.2023.3311510 |
| [26] | Dong, X., Bao, J., Chen, D., Zhang, W., Yu, N., Yuan, L., et al. (2022) CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, 18-24 June 2022, 12114-12124. https://doi.org/10.1109/cvpr52688.2022.01181 |
| [27] | Ouyang, D., He, S., Zhang, G., Luo, M., Guo, H., Zhan, J., et al. (2023) Efficient Multi-Scale Attention Module with Cross-Spatial Learning. ICASSP 2023—2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, 4-10 June 2023, 1-5. https://doi.org/10.1109/icassp49357.2023.10096516 |
| [28] | Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., et al. (2014) Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, 25-29 October 2014, 1724-1734. https://doi.org/10.3115/v1/d14-1179 |
| [29] | Chen, L., Cao, Y., Ma, L. and Zhang, J. (2020) A Deep Learning‐based Methodology for Precipitation Nowcasting with Radar. Earth and Space Science, 7, e2019EA000812. https://doi.org/10.1029/2019ea000812 |
| [30] | Denton, E. and Fergus, R. (2018) Stochastic Video Generation with a Learned Prior. 35th International Conference on Machine Learning (ICML 2018), Stockholm, 10-15 July 2018, 1906-1919. https://proceedings.mlr.press/v80/denton18a.html |
