|
|
一种带记忆的阻尼牛顿法
|
Abstract:
本文在引进阻尼技术的基础上构造了新的拟牛顿方程，该方程通过利用相邻两次迭代点之间的梯度差和变化量的线性组合，实现了曲率对的更新，以此改进校正公式。为求解高维问题，引入有限内存记忆框架，提出了D-LBFGS算法。利用D-LBFGS算法对神经网络中的权值阈值参数进行优化的算法，进一步提出了D-LBFGS—CNN算法，在解决无创光谱血糖浓度的预测问题中迭代351次时精度达到99.999%。所提出的D-LBFGS算法在一定程度上优于标准的LBFGS算法(迭代474次时精度99.999%)、阻尼牛顿法(精度99.359%)及文献中的LBFGS算法(精度99.744%)。
This paper constructs a new quasi-Newton equation based on the introduction of damping technology. This equation uses the linear combination of the gradient difference and change between two adjacent iteration points to update the curvature pair, thereby improving the correction formula. In order to solve high-dimensional problems, the limited memoryframe work is introduced and the D-LBFGS algorithm is proposed. The D-LBFGS algorithm is used to optimize the weight threshold parameters in the neural network, and the D-LBFGS—CNN algorithm is further proposed. The accuracy reaches 99.999% when iterating 351 times to solve the prediction problem of non-invasive spectral blood glucose concentration. The proposed D-LBFGS algorithm is superior to the standard LBFGS algorithm (accuracy 99.999% at 474 iterations), the damped Newton method (accuracy 99.359%), and the LBFGS algorithm in the literature (accuracy 99.744%) to a certain extent.
| [1] | Davidon, W.C. (1980) Conic Approximations and Collinear Scalings for Optimizers. SIAM Journal on Numerical Analysis, 17, 268-281. https://doi.org/10.1137/0717023 |
| [2] | Nocedal, J. (1980) Updating Quasi-Newton Matrices with Limited Storage. Mathematics of Computation, 35, 773-782. https://doi.org/10.1090/s0025-5718-1980-0572855-7 |
| [3] | Liu, D.C. and Nocedal, J. (1989) On the Limited Memory BFGS Method for Large Scale Optimization. Mathematical Programming, 45, 503-528. https://doi.org/10.1007/bf01589116 |
| [4] | Wang, X., Ma, S., Goldfarb, D. and Liu, W. (2017) Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization. SIAM Journal on Optimization, 27, 927-956. https://doi.org/10.1137/15m1053141 |
| [5] | Livieris, I.E., Tampakas, V. and Pintelas, P. (2018) A Descent Hybrid Conjugate Gradient Method Based on the Memoryless BFGS Update. Numerical Algorithms, 79, 1169-1185. https://doi.org/10.1007/s11075-018-0479-1 |
| [6] | 陈昱含. 求解大规模非凸无约束优化问题的两类修正L-BFGS方法[D]: [硕士学位论文]. 重庆: 重庆师范大学, 2021. |
| [7] | Berahas, A.S., Jahani, M., Richtárik, P. and Taká？, M. (2021) Quasi-Newton Methods for Machine Learning: Forget the Past, Just Sample. Optimization Methods and Software, 37, 1668-1704. https://doi.org/10.1080/10556788.2021.1977806 |
| [8] | Di Serafino, D., Toraldo, G. and Viola, M. (1980) Using Gradient Directions to Get Global Convergence of Newton-Type Methods. Applied Mathematics and Computation, 24, 111-117. |
| [9] | Andrei, N. (2021) A Note on Memory-Less SR1 and Memory-Less BFGS Methods for Large-Scale Unconstrained Optimization. Numerical Algorithms, 90, 223-240. https://doi.org/10.1007/s11075-021-01185-8 |
| [10] | Powell, M.J.D. (1978) Algorithms for Nonlinear Constraints That Use Lagrangian Functions. Mathematical Programming, 14, 224-248. https://doi.org/10.1007/bf01588967 |
| [11] | Nocedal, J. and Wright, S.J. (2006) Numerical Optimization. 2nd Edition, Springer, 32-36. |
| [12] | 赵睿颖. 用于深度学习的一种改进L-BFGS算法[J]. 首都师范大学学报(自然科学版), 2021, 42(5): 8-14. |
| [13] | Rumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986) Learning Representations by Back-Propagating Errors. Nature, 323, 533-536. https://doi.org/10.1038/323533a0 |
| [14] | 王朱宇. 基于机器学习的血糖光谱数据挖掘研究[D]. [硕士学位论文]. 长春: 长春理工大学, 2021. |
| [15] | Rafati, J. and Marica, R.F. (2020) Quasi-Newton Optimization Methods for Deep Learning Applications. Springer, 9-38. |
