The L^1-critical parabolic-elliptic Patlak-Keller-Segel system is a classical model of chemotactic aggregation in micro-organisms well-known to have critical mass phenomena. In this paper we study this critical mass phenomenon in the context of Patlak-Keller-Segel models with spatially varying diffusivity and decay rate of the chemo-attractant. The primary tool for the proof of global existence below the critical mass is the use of pseudo-differential operators to precisely evaluate the leading order quadratic portion of the potential energy (interaction energy). Under the assumption of radial symmetry, blow-up is proved above critical mass using a maximum-principle type argument based on comparing the mass distribution of solutions to a barrier consisting of the unique stationary solutions of the scale-invariant PKS. Although effective where standard Virial methods do not apply, this method seems to be dependent on the assumption of radial symmetry. For technical reasons we work in dimensions three and higher where L^1-critical variants of the PKS have porous media-type nonlinear diffusion on the organism density, resulting in finite speed of propagation and simplified functional inequalities.