|
|
基于Transformer的端到端深度符号回归方法
|
Abstract:
本研究提出了一种基于Transformer的端到端深度符号回归(TFSR)模型，用于通过自动生成数学表达式来解决符号回归问题。符号回归是一种无需先验模型形式的回归方法，旨在通过给定的数据推导出合理的数学表达式，具有较高的可解释性。随着深度学习的进步，深度符号回归(DSR)利用神经网络的强大数据处理能力，能够从高维数据中提取潜在的数学规律。本文提出的TFSR模型结合了进化算法和Transformer架构，通过自注意力机制有效处理数据中的长距离依赖关系，简化了符号回归过程并提升了效率与精度。此外，本研究还引入了前缀符号表示法和二叉树结构来高效表示和处理数学表达式，采用了分区采样方法生成均匀分布的采样数据。实验结果表明，TFSR模型在不同规模的数据集上具有良好的学习能力和泛化能力，尤其在处理复杂数学表达式时表现优异。通过基准测试与其他符号回归方法的对比，本研究的模型在复杂公式建模任务中展现了更强的表现，尤其在解决困难组问题时，TFSR模型表现出显著的优势。该研究为符号回归任务提供了一种新的端到端深度学习方法，尤其适用于复杂科学模型的自动构建，具有广泛的应用潜力。
This study proposes an end-to-end Transformer-based deep symbolic regression (TFSR) model to solve symbolic regression problems by automatically generating mathematical expressions. Symbolic regression is a regression method that does not require a priori model form. It aims to derive reasonable mathematical expressions from given data and has high interpretability. With the advancement of deep learning, deep symbolic regression (DSR) uses the powerful data processing capabilities of neural networks to extract potential mathematical laws from high-dimensional data. The TFSR model proposed in this paper combines evolutionary algorithms and Transformer architectures, effectively handles long-distance dependencies in data through self-attention mechanisms, simplifies the symbolic regression process, and improves efficiency and accuracy. In addition, this study also introduces prefix symbol representation and binary tree structure to efficiently represent and process mathematical expressions, and uses partition sampling methods to generate uniformly distributed sampling data. Experimental results show that the TFSR model has good learning and generalization capabilities on datasets of different sizes, especially when processing complex mathematical expressions. Through benchmark tests and comparisons with other symbolic regression methods, the model of this study shows stronger performance in complex formula modeling tasks, especially when solving difficult group problems, the TFSR model shows significant advantages. This study provides a new end-to-end deep learning method for symbolic regression tasks, which is particularly suitable for the automatic construction of complex scientific models and has broad application potential.
| [1] | Koza, J. (1994) Genetic Programming as a Means for Programming Computers by Natural Selection. Statistics and Computing, 4, 87-112. https://doi.org/10.1007/bf00175355 |
| [2] | Keijzer, M. (2003) Improving Symbolic Regression with Interval Arithmetic and Linear Scaling. In: Ryan, C., Soule, T., Keijzer, M., Tsang, E., Poli, R. and Costa, E., Eds., Lecture Notes in Computer Science, Springer, 70-82. https://doi.org/10.1007/3-540-36599-0_7 |
| [3] | Vladislavleva, E.J., Smits, G.F. and den Hertog, D. (2009) Order of Nonlinearity as a Complexity Measure for Models Generated by Symbolic Regression via Pareto Genetic Programming. IEEE Transactions on Evolutionary Computation, 13, 333-349. https://doi.org/10.1109/tevc.2008.926486 |
| [4] | McConaghy, T. (2011) FFX: Fast, Scalable, Deterministic Symbolic Regression Technology. In: Riolo, R., Vladislavleva, E. and Moore, J., Eds., Genetic and Evolutionary Computation, Springer, 235-260. https://doi.org/10.1007/978-1-4614-1770-5_13 |
| [5] | Schmidt, M. and Lipson, H. (2009) Symbolic Regression of Implicit Equations. In: Riolo, R., O’Reilly, UM. and McConaghy, T., Eds., Genetic and Evolutionary Computation, Springer, 73-85. https://doi.org/10.1007/978-1-4419-1626-6_5 |
| [6] | Gupta, R., Srivastava, D., Sahu, M., Tiwari, S., Ambasta, R.K. and Kumar, P. (2021) Artificial Intelligence to Deep Learning: Machine Intelligence Approach for Drug Discovery. Molecular Diversity, 25, 1315-1360. https://doi.org/10.1007/s11030-021-10217-3 |
| [7] | Angelis, D., Sofos, F. and Karakasidis, T.E. (2023) Artificial Intelligence in Physical Sciences: Symbolic Regression Trends and Perspectives. Archives of Computational Methods in Engineering, 30, 3845-3865. https://doi.org/10.1007/s11831-023-09922-z |
| [8] | Makke, N. and Chawla, S. (2024) Interpretable Scientific Discovery with Symbolic Regression: A Review. Artificial Intelligence Review, 57, Article No. 2. https://doi.org/10.1007/s10462-023-10622-0 |
| [9] | Rudin, C. (2019) Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Nature Machine Intelligence, 1, 206-215. https://doi.org/10.1038/s42256-019-0048-x |
| [10] | Lample, G. and Charton, F. (2019) Deep Learning for Symbolic Mathematics. arXiv:191201412. |
| [11] | Biggio, L., Bendinelli, T., Lucchi, A., et al. (2020) A Seq2Seq Approach to Symbolic Regression. Proceedings of the Learning Meets Combinatorial Algorithms at NeurIPS2020. |
| [12] | Wu, M., Li, W., Yu, L., et al. (2023) Discovering Mathematical Expressions through DeepSymNet: A Classification-Based Symbolic Regression Framework. IEEE Transactions on Neural Networks and Learning Systems. |
| [13] | Udrescu, S. and Tegmark, M. (2020) AI Feynman: A Physics-Inspired Method for Symbolic Regression. Science Advances, 6, eaay2631. https://doi.org/10.1126/sciadv.aay2631 |
| [14] | Udrescu, S.-M., Tan, A., Feng, J., et al. (2020) AI Feynman 2.0: Pareto-Optimal Symbolic Regression Exploiting Graph Modularity. Advances in Neural Information Processing Systems, 33, 4860-4871. |
| [15] | Kamienny, P.-A., D’ascoli, S., Lample, G., et al. (2022) End-to-End Symbolic Regression with Transformers. Advances in Neural Information Processing Systems, 35, 10269-10281. |
| [16] | Lalande, F., Matsubara, Y., Chiba, N., et al. (2023) A Transformer Model for Symbolic Regression towards Scientific Discovery. arXiv:231204070. |
| [17] | Vastl, M., Kulhánek, J., Kubalík, J., Derner, E. and Babuška, R. (2024) SymFormer: End-to-End Symbolic Regression Using Transformer-Based Architecture. IEEE Access, 12, 37840-37849. https://doi.org/10.1109/access.2024.3374649 |
| [18] | Matsubara, Y., Chiba, N., Igarashi, R., et al. (2022) Rethinking Symbolic Regression Datasets and Benchmarks for Scientific Discovery. arXiv:220610540. |
