The Composite Cosine Transform on the Stiefel Manifold and Generalized Zeta Integrals

We introduce a new integral transform $T^\lam f$, $\lam \in C^m$, on the Stiefel manifold of orthonormal $m$-frames in $R^n$ which generalizes the $\lam$-cosine transform on the Grassmann manifold of $m$-dimensional linear subspaces of $R^n$. We call it the composite cosine transform, by taking into account that its kernel agrees with the composite power function of the cone of positive definite symmetric matrices. Our aim is to describe the set of all $\lam \in C^m$ for which $T^\lam$ is injective on the space of integrable functions. We obtain the precise description of this set in some important cases, in particular, for $\lam$-cosine transforms on Grassmann manifolds. The main tools are the classical Fourier analysis of functions of matrix argument and the relevant zeta integrals.