Ultra-high cosmic rays (UHECRs) with energies >10^19 eV emitted at cosmological distances will be attenuated by cosmic microwave and infrared background radiation through photohadronic processes. Lower energy extra-galactic cosmic rays (~10^18-10^19 eV) can only travel a linear distance smaller than ~Gpc in a Hubble time due to the diffusion if the extra-galactic magnetic fields are as strong as nano Gauss. These prevent us from directly observing most of the UHECRs in the universe, and thus the observed UHECR intensity reflects only the emissivity in the nearby universe within hundreds of Mpc. However, UHECRs in the distant universe, through interactions with the cosmic background photons, produce UHE electrons and gamma-rays that in turn initiate electromagnetic cascades on cosmic background photons. This secondary cascade radiation forms part of the extragalactic diffuse GeV-TeV gamma-ray radiation and, unlike the original UHECRs, is observable. Motivated by new measurements of extragalactic diffuse gamma-ray background radiation by Fermi/LAT, we obtained upper limits placed on the UHECR emissivity in the distant universe by requiring that the cascade radiation they produce not exceed the observed levels. By comparison with the gamma-ray emissivity of candidate UHECR sources (such as GRBs and AGNs) at high-redshifts, we find that the obtained upper limit for a flat proton spectrum is ~10^1.5 times larger than the gamma-ray emissivity in GRBs and ~10 times smaller than the gamma-ray emissivity in BL Lac objects. In the case of iron nuclei composition, the derived upper limit of the UHECR emissivity is a factor of 3-5 times higher. Robust upper limit on the cosmogenic neutrino flux is further obtained, which is marginally reachable by the Icecube detector and the next-generation detector JEM-EUSO.