The Rectified Linear Unit (ReLU) activation function is widely employed in deep learning (DL). ReLU shares structural similarities with censored regression and Tobit models common in econometrics and statistics. Although the conventional Tobit maximum likelihood estimator (CTMLE) is frequently applied in these fields, the insights gained have not been fully incorporated into DL research. When models are affected by random observation errors or distributional misspecification, CTMLE often exhibits substantial bias. To address this limitation, we consider the Grouping Adjusted Median Estimator (GAME). GAME is a robust method that does not rely on specific distributional assumptions and is constructed through three stages: grouping, adjustment, and computation of adjusted medians combined with weighted Tobit maximum likelihood estimation. Monte Carlo experiments show that GAME outperforms CTMLE in non-standard settings while incurring only minor efficiency losses under standard conditions. Max pooling is widely used in DL, and its mechanism resembles grouping. This study also evaluates the max pooling estimator (MPE). However, MPE often performs poorly, as the estimated boundaries between target and background regions may be reversed relative to the true ones. While max pooling can be useful for detecting weak signals, it may yield misleading results in high-noise situations, necessitating special care. GAME offers a promising alternative for mitigating noise effects, and the median can be generalized to arbitrary percentiles, providing additional flexibility in estimation.
References
[1]
Bai, Y. (2022) ReLU-function and Derived Function Review. SHS Web of Conferences, 144, Article 02006. https://doi.org/10.1051/shsconf/202214402006
[2]
Li, G.H.Y., Sekine, R., Nehra, R., Gray, R.M., Ledezma, L., Guo, Q., et al. (2022) All-optical Ultrafast ReLU Function for Energy-Efficient Nanophotonic Deep Learning. Nanophotonics, 12, 847-855. https://doi.org/10.1515/nanoph-2022-0137
[3]
Nworu, C.C., Ekpenyong, E.J., Chisimkwuo, J., Onyeukwu, C.N., Okwara, G. and Agwu, O.J. (2022) The Effects of Modified ReLU Activation Functions in Image Classification. Biomedical Engineering and Medical Devices, 7, Article 1000237.
[4]
Cabanilla, K.I.M., Mohammad, R.Z. and Lope, J.E.C. (2024) Neural Networks with ReLU Powers Need Less Depth. Neural Networks, 172, Article 106073. https://doi.org/10.1016/j.neunet.2023.12.027
[5]
Hu, P., Sun, L., Hu, C., Dai, L., Guo, S. and Yu, M. (2024) DReP: Deep ReLU Pruning for Fast Private Inference. Journal of Systems Architecture, 152, Article 103156. https://doi.org/10.1016/j.sysarc.2024.103156
[6]
Sooksatra, K., Hamerly, G. and Rivas, P. (2023) Is ReLU Adversarially Robust? arXiv:2405.03777.
[7]
Ahmadnejad, A., Assadi, M.M. and Koohi, S. (2025) All-Optical Doubly Resonant Cavities for ReLU Function in Nanophotonic Deep Learning. arXiv:2504.19692v1.
[8]
Horuz, C.C., Kasenbacher, G., Higuchi, S., Kairat, S., Stoltz, J., Pesl, M., et al. (2025) The Resurrection of the ReLU. arXiv:2505.22074v1.
[9]
Mestre, J.I., Barrachina, S., Quezada, D. and Dolz, M.F. (2025) Deep Learning Inference Optimisation for IoT: Conv2D-ReLU-BN Layer Fusion and Quantisation. The Journal of Supercomputing, 81, Article No. 621. https://doi.org/10.1007/s11227-025-07107-y
[10]
Ullah, S., Javed, A., Aljasem, M. and Saudagar, A.K.J. (2025) Eff-ReLU-Net: A Deep Learning Framework for Multiclass Wound Classification. BMC Medical Imaging, 25, Article No. 257. https://doi.org/10.1186/s12880-025-01785-z
[11]
Ueno, Y., Hironaka, Y., Yoshikawa, N. and Yamanashi, Y. (2025) High-Speed, Ultra-Low-Power, and Robust Superconductive Neuron with ReLU Activation. Neuromorphic Computing and Engineering, 5, Article 044003. https://doi.org/10.1088/2634-4386/ae0aee
[12]
Frei, S., Vardi, G., Bartlett, P.L., Srebro, N. and Hu, W. (2022) Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data. arXiv:2210.07082.
[13]
Mujhid, A., Surono, S., Irsalinda, N. and Thobirin, A. (2022) Comparison and Combination of Leaky ReLU and ReLU Activation Function and Three Optimizers on Deep CNN for COVID-19 Detection. In: Tallón-Ballesteros, A.J., Ed., Frontiers in Artificial Intelligence and Applications, IOS Press, 50-57. https://doi.org/10.3233/faia220369
[14]
Mary Shyni, H. and Chitra, E. (2024) PulmonNet V1: Leveraging the Benefit of Leaky ReLU Activation for the Local and Multi-Scale Global Feature Integration of Chest Radiographs to Classify Pulmonary Diseases. Biomedical Signal Processing and Control, 96, Article 106600. https://doi.org/10.1016/j.bspc.2024.106600
[15]
Singh, A.K. and Kumar, A. (2025) Multi-Objective: Hybrid Particle Swarm Optimization with Firefly Algorithm for Feature Selection with Leaky ReLU. Discover Artificial Intelligence, 5, Article No. 192. https://doi.org/10.1007/s44163-025-00428-0
[16]
Greene, W.H. (2020) Econometric Analysis. 8th and International Edition, Peason Education.
[17]
Carson, R.T. and Sun, Y. (2007) The Tobit Model with a Non‐Zero Threshold. The Econometrics Journal, 10, 488-502. https://doi.org/10.1111/j.1368-423x.2007.00218.x
[18]
Amore, M.D. and Murtinu, S. (2019) Tobit Models in Strategy Research: Critical Issues and Applications. Global Strategy Journal, 11, 331-355. https://doi.org/10.1002/gsj.1363
[19]
Xue, L., Qu, A., Guo, X. and Hao, C. (2024) Research on Environmental Performance Measurement and Influencing Factors of Key Cities in China Based on Super-Efficiency SBM-Tobit Model. Sustainability, 16, Article 4792. https://doi.org/10.3390/su16114792
[20]
Tobin, J. (1958) Estimation of Relationships for Limited Dependent Variables. Econometrica, 26, 24-36. https://doi.org/10.2307/1907382
[21]
Gholamalinezhad, H. and Khosravi, H. (2020) Pooling Methods in Deep Neural Networks, a Review. arXiv:2009.07485.
[22]
Saeedan, F., Weber, N., Goesele, M. and Roth, S. (2018) Detail-Preserving Pooling in Deep Networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 9108-9116. https://doi.org/10.1109/cvpr.2018.00949
[23]
Zafar, A., Aamir, M., Mohd Nawi, N., Arshad, A., Riaz, S., Alruban, A., et al. (2022) A Comparison of Pooling Methods for Convolutional Neural Networks. Applied Sciences, 12, Article 8643. https://doi.org/10.3390/app12178643
[24]
Baniata, L.H., Kang, S., Alsharaiah, M.A. and Baniata, M.H. (2024) Advanced Deep Learning Model for Predicting the Academic Performances of Students in Educational Institutions. Applied Sciences, 14, Article 1963. https://doi.org/10.3390/app14051963
[25]
Hosny, K.M., Magdi, A., ElKomy, O. and Hamza, H.M. (2024) Digital Image Watermarking Using Deep Learning: A Survey. Computer Science Review, 53, Article 100662. https://doi.org/10.1016/j.cosrev.2024.100662
[26]
Zhao, L. and Zhang, Z. (2024) A Improved Pooling Method for Convolutional Neural Networks. Scientific Reports, 14, Article No. 1589. https://doi.org/10.1038/s41598-024-51258-6
[27]
Murray, N. and Perronnin, F. (2014) Generalized Max Pooling. 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, 23-28 June 2014, 2473-2480. https://doi.org/10.1109/cvpr.2014.317
[28]
Brutzkus, A. and Globerson, A. (2021) An Optimization and Generalization Analysis for Max-Pooling Networks. Proceedings of Machine Learning Research, 161, 1650-1660.
[29]
Matoba, K., Dimitriadis, N. and Fleuret, F. (2023) Benefits of Max Pooling in Neural Networks: Theoretical and Experimental Evidence. Transactions on Machine Learning Research.
[30]
Arulananth, T.S., Prakash, S.W., Ayyasamy, R.K., Kavitha, V.P., Kuppusamy, P.G. and Chinnasamy, P. (2024) Classification of Paediatric Pneumonia Using Modified Densenet-121 Deep-Learning Model. IEEE Access, 12, 35716-35727. https://doi.org/10.1109/access.2024.3371151
[31]
Mian, X., Bingtao, Z., Shiqiang, C. and Song, L. (2024) MCMP-Net: MLP Combining Max Pooling Network for sEMG Gesture Recognition. Biomedical Signal Processing and Control, 90, Article 105846. https://doi.org/10.1016/j.bspc.2023.105846
[32]
Kohler, M. and Langer, S. (2025) Statistical Theory for Image Classification Using Deep Convolutional Neural Network with Cross-Entropy Loss under the Hierarchical Max-Pooling Model. Journal of Statistical Planning and Inference, 234, Article 106188. https://doi.org/10.1016/j.jspi.2024.106188
[33]
Song, W., Dai, S., Huang, D., Song, J. and Antonio, L. (2021) Median-Pooling Grad-Cam: An Efficient Inference Level Visual Explanation for CNN Networks in Remote Sensing Image Classification. In: Lokoč, J., et al., Eds., Lecture Notes in Computer Science, Springer International Publishing, 134-146. https://doi.org/10.1007/978-3-030-67835-7_12
[34]
Nawata, K. (1990) Robust Estimation Based on Grouped-Adjusted Data in Censored Regression Models. Journal of Econometrics, 43, 337-362. https://doi.org/10.1016/0304-4076(90)90124-c
[35]
Bishop, C.M. and Bishop, H. (2024) Deep Learning: Foundation and Concepts. Springer.
[36]
Segeth, K. (2021) Multivariate Data Fitting Using Polyharmonic Splines. Journal of Computational and Applied Mathematics, 397, Article 113651. https://doi.org/10.1016/j.cam.2021.113651
[37]
Dantony, E., Uhry, Z., Fauvernier, M., Coureau, G., Mounier, M., Trétarre, B., et al. (2024) Multidimensional Penalized Splines for Survival Models: Illustration for Net Survival Trend Analyses. International Journal of Epidemiology, 53, dyae033. https://doi.org/10.1093/ije/dyae033
[38]
Ameniya, T. (1985) Advanced Econometrics. Harvard University Press.
[39]
Nawata, K. (1990) Robust Estimation Based on Grouped-Adjusted Data in Linear Regression Models. Journal of Econometrics, 43, 317-336. https://doi.org/10.1016/0304-4076(90)90123-b
[40]
Nawata, K. (1994) Estimation of the Boundary of the Two Regions by the Grouping Method. Journal of the Japan Statistical Society, 24, 14-35.
[41]
Nawata, K. (2022) An Analysis of Two-Dimensional Image Data Using a Grouping Estimator. Open Journal of Statistics, 12, 33-48. https://doi.org/10.4236/ojs.2022.121003
[42]
Bieder, F., Sandkühler, R. and Cattin, P.C. (2021) Comparison of Methods Generalizing Max-and Average-Pooling. arXiv:2103.01746.
[43]
Yang, J., Xie, F., Fan, H., Jiang, Z. and Liu, J. (2018) Classification for Dermoscopy Images Using Convolutional Neural Networks Based on Region Average Pooling. IEEE Access, 6, 65130-65138. https://doi.org/10.1109/access.2018.2877587
[44]
Lee, C., Shih, J., Su, W., Lien, C. and Han, C. (2024) Combination of Global Maximum Pooling and Local Average Pooling for Unsupervised Fine-Grained Image Retrieval. Proceedings of the 2024 7th Artificial Intelligence and Cloud Computing Conference, Tokyo, 14-16 December 2024, 299-307. https://doi.org/10.1145/3719384.3719427
[45]
Özdemir, C. (2023) Avg-Topk: A New Pooling Method for Convolutional Neural Networks. Expert Systems with Applications, 223, Article 119892. https://doi.org/10.1016/j.eswa.2023.119892
