The generalized linear model (GLM) is widely used in econometrics and statistics. In a GLM, a location parameter of the dependent variable—typically the expected value or mean—is linked to a linear combination of independent variables through a link function. The choice of link function is determined by the distribution of the error term. In contrast, activation functions are used in the convolutional layers of deep learning (DL) models. Common activation functions include the indicator (step) function, sigmoid, rectified linear unit (ReLU), and leaky ReLU. Although the universal approximation theorem shows that DL models using these activation functions can approximate a wide class of functions, major challenges remain, including overfitting, local minima, and substantial computational requirements. It is therefore essential to select an appropriate activation function. Link functions in GLMs and activation functions in DL are closely related and often coincide. In this note, we examine the similarities between these functions and propose an activation function derived from the truncated mean. We focus on the theoretical and methodological aspects rather than practical applications. The relationship between link functions and activation functions remains conjectural and warrants further research.
References
[1]
McCullagh, P. and Nelder, J.A. (1989) Generalized Linear Models. Second Edition, Chapman & Hall.
[2]
Olsson, U. (2002) Generalized Linear Models, an Applied Approach. Studentlitteratur.
[3]
Islam, M.A. and Biswas, S.C. (2025) Generalized Linear Models and Extensions. Springer Nature.
[4]
Wood, S.N. (2025) Generalized Additive Models. Annual Review of Statistics and Its Application, 12, 497-526. https://doi.org/10.1146/annurev-statistics-112723-034249
[5]
Correia, S., Guimar？es, P. and Zylkin, T. (2026) Verifying the Existence of Maximum Likelihood Estimates for Generalized Linear Models. Econometric Reviews, 45, 1270-1305. https://doi.org/10.1080/07474938.2026.2673989
[6]
Dubey, S.R., Singh, S.K. and Chaudhuri, B.B. (2022) Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark. Neurocomputing, 503, 92-108. https://doi.org/10.1016/j.neucom.2022.06.111
[7]
Bishop, C.M. and Bishop, H. (2024) Deep Learning: Foundation and Concepts. Springer.
[8]
Nawata, K. (1990) Robust Estimation Based on Grouped-Adjusted Data in Censored Regression Models. Journal of Econometrics, 43, 337-362. https://doi.org/10.1016/0304-4076(90)90124-c
[9]
Nawata, K. (2026) Estimating the Rectified Linear Unit Activation Function in Deep Learning Using the Grouping-Adjusted Median Estimator: An Evaluation of Max Pooling. Open Journal of Statistics, 16, 1-15. https://doi.org/10.4236/ojs.2026.161001
[10]
Behboodi, A., Rauhut, H. and Schnoor, E. (2020) Generalization Bounds for Deep Thresholding Networks. https://arxiv.org/abs/2010.15658v1
[11]
Imaizumi, M. and Schmidt-Hieber, J. (2023) On Generalization Bounds for Deep Networks Based on Loss Surface Implicit Regularization. IEEE Transactions on Information Theory, 69, 1203-1223. https://doi.org/10.1109/tit.2022.3215088
[12]
Ergen, T., Gulluk, H.I., Lacotte, J. and Pilanci, M. (2023) Globally Optimal Training of Neural Networks with Threshold Activation Functions. https://arxiv.org/abs/2303.03382
[13]
Dhayalkar, S.R. (2025) Neural Networks as Universal Finite-State Machines: A Constructive Deterministic Finite Automaton Theory. https://arxiv.org/abs/2505.11694
[14]
Rumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986) Learning Representations by Back-Propagating Errors. Nature, 323, 533-536. https://doi.org/10.1038/323533a0
[15]
Roodschild, M., Gotay？Sardi？as, J. and Will, A. (2020) A New Approach for the Vanishing Gradient Problem on Sigmoid Activation. Progress in Artificial Intelligence, 9, 351-360. https://doi.org/10.1007/s13748-020-00218-y
[16]
Mulindwa, D.B. and Du, S. (2023) An N-Sigmoid Activation Function to Improve the Squeeze-And-Excitation for 2D and 3D Deep Networks. Electronics, 12, Article No. 911. https://doi.org/10.3390/electronics12040911
[17]
Huang, H., Feng, S., Li, J., Guan, T. and Zhu, H. (2025) Adaptive Belief Rule Base Modeling of Complex Industrial Systems Based on Sigmoid Functions. Entropy, 27, Article No. 1157. https://doi.org/10.3390/e27111157
[18]
Bai, Y. (2022) ReLU-Function and Derived Function Review. SHS Web of Conferences, 144, Article No. 02006. https://doi.org/10.1051/shsconf/202214402006
[19]
Nworu, C.C., Ekpenyong, E.J., Chisimkwuo, J., Onyeukwu, C.N., Okwara, G. and Agwu, O.J. (2022) The Effects of Modified ReLU Activation Functions in Image Classification. Biomedical Engineering and Medical Devices, 7, Article ID: 1000237.
[20]
Sooksatra, K., Hamerly, G. and Rivas, P. (2024) Is ReLU Adversarially Robust? https://arxiv.org/abs/2405.03777
[21]
Zhang, S., Lu, J. and Zhao, H. (2024) Deep Network Approximation: Beyond ReLU to Diverse. Activation Functions Journal of Machine Learning Research, 25, 1-39.
[22]
Horuz, C.C., Kasenbacher, G., Higuchi, S., Kairat, S., Stoltz, J., Pesl, M., et al. (2025) The Resurrection of the ReLU. https://arxiv.org/abs/2505.22074
[23]
Agarap, A.F. (2026) Deep Learning Using Rectified Linear Units (ReLU). https://arxiv.org/abs/1803.08375
[24]
Maniatopoulos, A. and Mitianoudis, N. (2021) Learnable Leaky ReLU (LeLeLU): An Alternative Accuracy-Optimized Activation Function. Information, 12, Article No. 513. https://doi.org/10.3390/info12120513
[25]
Mastromichalakis, S. (2021) ALReLU: A Different Approach on Leaky ReLU Activation Function to Improve Neural Networks Performance. https://arxiv.org/abs/2012.07564
[26]
Frei, S., Vardi, G., Bartlett, P.L., Srebro, N. and Hu, W. (2022) Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data. https://arxiv.org/abs/2210.07082
[27]
Mujhid, A., Surono, S., Irsalinda, N. and Thobirin, A. (2022) Comparison and Combination of Leaky ReLU and ReLU Activation Function and Three Optimizers on Deep CNN for COVID-19 Detection. In: Tallón-Ballesteros, A.J., Ed., Fuzzy Systems and Data Mining VIII, IOS Press, 50-57. https://doi.org/10.3233/faia220369
[28]
Neufeld, A. and Nguyen, T.A. (2026) Multilevel Picard Approximations and Deep Neural Networks with ReLU, Leaky ReLU, and Softplus Activation Overcome the Curse of Dimensionality When Approximating Semilinear Parabolic Partial Differential Equations in Lp-Sense. Journal of Computational and Applied Mathematics, 487, Article ID: 117736. https://doi.org/10.1016/j.cam.2026.117736
[29]
Tikk, D., Kóczy, L.T. and Gedeon, T.D. (2003) A Survey on Universal Approximation and Its Limits in Soft Computing Techniques. International Journal of Approximate Reasoning, 33, 185-202. https://doi.org/10.1016/s0888-613x(03)00021-5
[30]
Kidger, P. and Lyons, T. (2020) Universal Approximation with Deep Narrow Networks. Proceedings of Machine Learning Research, 125, 1-22.
[31]
Augustine, M.T. (2024) A Survey on Universal Approximation Theorems. https://arxiv.org/abs/2407.12895
[32]
Kratsios, A., Cheng, T.S. and Roy, D. (2025) Beyond Universal Approximation Theorems: Algorithmic Uniform Approximation by Neural Networks Trained with Noisy Data. https://arxiv.org/abs/2509.00924
[33]
Nawata, K. (1997) Estimation of Generalised Regression Models by the Grouping Method. Mathematics and Computers in Simulation, 43, 503-510. https://doi.org/10.1016/s0378-4754(97)00038-4
[34]
Nawata, K. (1989) Semiparametric Estimation and Efficiency Bounds of Binary Choice Models When the Models Contain One Continuous Variable. Economics Letters, 31, 21-26. https://doi.org/10.1016/0165-1765(89)90105-5
Nawata, K. (1994) Notes on Estimation of the Tobit Models by Powell’s Least Absolute Deviations Estimator. Economic Studies Quarterly, 45, 339-346.
[37]
Lin, W., Piao, Z. and Fung, C.C.A. (2025) Delving into Unsupervised Hebbian Learning from Artificial Intelligence Perspectives. Machine Learning and Knowledge Extraction, 7, Article No. 143. https://doi.org/10.3390/make7040143
[38]
Ramachandran, P., Zoph, B. and Le, B.Q.V. (2017) Searching for Activation Functions. https://arxiv.org/abs/1710.05941
[39]
Zhao, H., Liu, F., Li, L. and Luo, C. (2018) A Novel Softplus Linear Unit for Deep Convolutional Neural Networks. Applied Intelligence, 48, 1707-1720. https://doi.org/10.1007/s10489-017-1028-7
[40]
Hendrycks, D. and Gimpel, K. (2016) Gaussian Error Linear Units (GELUs). https://arxiv.org/abs/1606.08415
[41]
Amemiya, T. (1985) Advanced Econometrics. Harvard University Press.
[42]
Heckman, J.J. (1976) The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models. Annals of Economic and Social Measurement, 5, 475-492.
[43]
Heckman, J.J. (1979) Sample Selection Bias as a Specification Error. Econometrica, 47, 153-161. https://doi.org/10.2307/1912352
[44]
Nawata, K. (1993) A Note on the Estimation of Models with Sample-Selection Biases. Economics Letters, 42, 15-24. https://doi.org/10.1016/0165-1765(93)90167-b
[45]
Nawata, K. (1994) Estimation of Sample Selection Bias Models by the Maximum Likelihood Estimator and Heckman’s Two-Step Estimator. Economics Letters, 45, 33-40. https://doi.org/10.1016/0165-1765(94)90053-1
[46]
Nawata, K. and Nagase, N. (1996) Estimation of Sample Selection Bias Models. Econometric Reviews, 15, 387-400. https://doi.org/10.1080/07474939608800363
