Dispersion Is Not a Nuisance: Penalized Joint Optimal Estimating Functions for Over-Dispersed Regression

Penalized regression is often assessed on how well it recovers the mean model. On the other hand, the dispersion parameter is treated as a nuisance quantity estimated from the residuals that the penalty has already altered. This paper asks what penalization does to dispersion inference itself. Working within a joint Optimal Estimating Function (OEF) system that estimates regression and dispersion parameters simultaneously from orthogonalized moment conditions, we apply Ridge, LASSO, Adaptive LASSO and SCAD penalties to the regression equation only leaving the dispersion equation unpenalized. Consequently, we examine what this does to the intra-cluster correlation of the Beta-Binomial model and the heterogeneity parameter of the Negative-Binomial model. Across 500 Monte Carlo replications per scenario in a factorial design crossing over-dispersion level with signal density, penalization disturbs dispersion estimation substantially more than it disturbs regression coefficient estimation. This mechanism is the mean-dispersion coupling of the joint OEF system. Shrinkage of coefficients alters the residual structure from which the dispersion parameter is estimated and since the dispersion equation is unpenalized, the estimate absorbs the mean model error. The disturbance takes a different form in each model. Under the Beta-Binomial, Ridge over-shrinkage drives a compensation cycle that inflates the dispersion estimate and produces multimodal sampling distributions. Under the Negative Binomial, the exponential mean propagates coefficient error quadratically into the variance function and the L1-type penalties underestimate dispersion severely. A log-scale interval raises Negative Binomial dispersion coverage by up to 55 percentage points while the analogous logit-scale interval does almost nothing for the Beta-Binomial, where $\widehat{\rho }$ is bounded and already close to symmetric. We also show that the Pearson moment estimator attains better nominal dispersion coverage than the Godambe-optimal OEF in the Beta-Binomial design and examine why an asymptotically optimal construction can undercover at $n=100$ sample size.