The integration of artificial intelligence into the development and production of mechatronic products offers a substantial opportunity to enhance efficiency, adaptability, and system performance. This paper examines the utilization of reinforcement learning as a control strategy, with a particular focus on its deployment in pivotal stages of the product development lifecycle, specifically between system architecture and system integration and verification. A controller based on reinforcement learning was developed and evaluated in comparison to traditional proportional-integral controllers in dynamic and fault-prone environments. The results illustrate the superior adaptability, stability, and optimization potential of the reinforcement learning approach, particularly in addressing dynamic disturbances and ensuring robust performance. The study illustrates how reinforcement learning can facilitate the transition from conceptual design to implementation by automating optimization processes, enabling interface automation, and enhancing system-level testing. Based on the aforementioned findings, this paper presents future directions for research, which include the integration of domain-specific knowledge into the reinforcement learning process and the validation of this process in real-world environments. The results underscore the potential of artificial intelligence-driven methodologies to revolutionize the design and deployment of intelligent mechatronic systems.
References
[1]
Åström, K.J. and Hägglund, T. (2006) Advanced PID Control. ISA—The Instrumentation, Systems, and Automation Society.
[2]
Lloyds Raja, G. and Ali, A. (2021) New PI-PD Controller Design Strategy for Industrial Unstable and Integrating Processes with Dead Time and Inverse Response. Journal of Control, Automation and Electrical Systems, 32, 266-280. https://doi.org/10.1007/s40313-020-00679-5
[3]
Jiménez, G.A., de la Escalera Hueso, A. and Gómez-Silva, M.J. (2023) Reinforcement Learning Algorithms for Autonomous Mission Accomplishment by Unmanned Aerial Vehicles: A Comparative View with DQN, SARSA and A2C. Sensors, 23, Article 9013. https://doi.org/10.3390/s23219013
[4]
De La Fuente, N. and Guerra, D.A.V. (2024) A Comparative Study of Deep Reinforce-ment Learning Models: DQN vs PPO vs A2C. arXiv: 2407.14151.
[5]
Vouros, G.A. (2023) Explainable Deep Reinforcement Learning: State of the Art and Challenges. arXiv: 2301.09937.
[6]
Sutton, R.S. and Barto, A. (2020) Reinforcement Learning: An Introduction. Second Edition, The MIT Press.
[7]
Gräßler, I. (2021) VDI/VDE 2206: Entwicklung mechatronischer und cyber-physischer Systeme. Inhaltsverzeichnis.
[8]
Gatti, C. (2015) Design of Experiments for Reinforcement Learning. Springer.
[9]
Ladosz, P., Weng, L., Kim, M. and Oh, H. (2022) Exploration in Deep Reinforcement Learning: A Survey. Information Fusion, 85, 1-22. https://doi.org/10.1016/j.inffus.2022.03.003
[10]
Fazdi, M.F. and Hsueh, P. (2023) Parameters Identification of a Permanent Magnet DC Motor: A Review. Electronics, 12, Article 2559. https://doi.org/10.3390/electronics12122559
[11]
Shah, R. and Sands, T. (2021) Comparing Methods of DC Motor Control for UUVs. Applied Sciences, 11, Article 4972. https://doi.org/10.3390/app11114972
[12]
Aribowo, W., Supari, S. and Suprianto, B. (2022) Optimization of PID Parameters for Controlling DC Motor Based on the Aquila Optimizer Algorithm. International Journal of Power Electronics and Drive Systems (IJPEDS), 13, 216-222. https://doi.org/10.11591/ijpeds.v13.i1.pp216-222
[13]
Spring, E. (2009) Elektrische Maschinen: Eine Einführung. Springer.
[14]
Borase, R.P., Maghade, D.K., Sondkar, S.Y. and Pawar, S.N. (2020) A Review of PID Control, Tuning Methods and Applications. International Journal of Dynamics and Control, 9, 818-827. https://doi.org/10.1007/s40435-020-00665-4
[15]
Chau, P.C. (2002) Process Control. Cambridge University Press. https://doi.org/10.1017/cbo9780511813665
[16]
Zhang, A., McAllister, R., Calandra, R., Gal, Y. and Levine, S. (2020) Learning Invariant Representations for Reinforcement Learning without Reconstruction. arXiv: 2006.10742.
[17]
Schwarzer, M., Anand, A., Goel, R., Hjelm, R.D., Courville, A. and Bachman, P. (2020) Data-Efficient Reinforcement Learning with Self-Predictive Representations. arXiv: 2007.05929.
[18]
Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., et al. (2020) Scipy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17, 261-272. https://doi.org/10.1038/s41592-019-0686-2
[19]
Pinto, V., Gonçalves, J. and Costa, P. (2020) Modeling and Control of a DC Motor Coupled to a Non-Rigid Joint. Applied System Innovation, 3, Article 24. https://doi.org/10.3390/asi3020024
[20]
Wang, Y. and Shao, H. (2000) Optimal Tuning for PI Controller. Automatica, 36, 147-152. https://doi.org/10.1016/s0005-1098(99)00130-2
[21]
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., et al. (2024) Deep Reinforcement Learning: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 35, 5064-5078. https://doi.org/10.1109/tnnls.2022.3207346
[22]
Matsuo, Y., LeCun, Y., Sahani, M., Precup, D., Silver, D., Sugiyama, M., et al. (2022) Deep Learning, Reinforcement Learning, and World Models. Neural Networks, 152, 267-275. https://doi.org/10.1016/j.neunet.2022.03.037
[23]
Arulkumaran, K., Deisenroth, M.P., Brundage, M. and Bharath, A.A. (2017) Deep Reinforcement Learning: A Brief Survey. IEEE Signal Processing Magazine, 34, 26-38. https://doi.org/10.1109/msp.2017.2743240
[24]
Haarnoja, T., Zhou, A., Abbeel, P. and Levine, S. (2018) Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv: 1801.01290.
[25]
Zhang, K., Wang, Z., Chen, G., Zhang, L., Yang, Y., Yao, C., et al. (2022) Training Effective Deep Reinforcement Learning Agents for Real-Time Life-Cycle Production Optimization. Journal of Petroleum Science and Engineering, 208, Article ID: 109766. https://doi.org/10.1016/j.petrol.2021.109766
[26]
Laskin, M., et al. (2021) URLB: Unsupervised Reinforcement Learning Benchmark. arXiv: 2110.15191.
[27]
Eschmann, J. (2021) Reward Function Design in Reinforcement Learning. In: Belousov, B., Abdulsamad, H., Klink, P., Parisi, S. and Peters, J., Eds., Reinforcement Learning Algorithms: Analysis and Applications, Springer, 25-33. https://doi.org/10.1007/978-3-030-41188-6_3
[28]
Kwon, M., Xie, S.M., Bullard, K. and Sadigh, D. (2023) Reward Design with Language Models. arXiv: 2303.00001.
[29]
Kiran, M. and Ozyildirim, M. (2022) Hyperparameter Tuning for Deep Reinforcement Learning Applications. arXiv: 2201.11182.
[30]
Felten, F., Gareev, D., Talbi, E.G. and Danoy, G. (2023) Hyperparameter Optimization for Multi-Objective Reinforcement Learning. arXiv: 2310.16487.
[31]
Dean, A., Voss, D. and Draguljić, D. (2017) Design and Analysis of Experiments. Springer.
[32]
Nüssgen, A., et al. (2023) Intelligent Component Manufacturability Testing in Virtual Product Development. Artificial Intelligence und Machine Learning in der CAE-Basierten Simulation, München, 23-24 Oktober 2023, 14-22.
