Aims Populus cathayana, a typical riparian tree in northern China, can produce substantial seeds and numerous clones. Several studies in the last few years indicated that the proportion of seedlings and clones in the population was correlated with factors such as river flooding, soil substrates and slope gradient. Our objective was to investigate the regeneration mode of Populus cathayana located in high-elevation woodland of Xiaowutai Mountains, revealing why there was such a phenomenon. Methods From 2010 to 2011, we excavated 1 284 small P. cathayana saplings along an elevational gradient in high-elevation woodland of Xiaowutai Mountains to determine their method of origin. The influence of allelopathy and mechanical effects of litter layer on seedling regeneration was investigated by a series of controlled experiments. Important findings Regeneration was overwhelmingly from clonal propagation. Litter layer had a negative effect on recruitment from seed. Under treatment of different concentrations of litter layer aqueous extracts, seed germination exhibited significant differences. Compared with the control treatment, 50 mg·mL–1 solution decreased seed germination by 41% and root length by 95%, and 100 mg·mL–1 solution completely inhibited germination. Furthermore, litter layer inhibited roots of seedlings from absorbing water and rapidly caused seedling death. Therefore, litter layer prevents seedling recruitment, and clonal propagation was the only effective regeneration mode of P. cathayana in the population.