We demonstrate that 60 keV electron irradiation drives the diffusion of threefold coordinated Si dopants in graphene by one lattice site at a time. First principles simulations reveal that each step is caused by an electron impact on a C atom next to the dopant. Although the atomic motion happens below our experimental time resolution, stochastic analysis of 38 such lattice jumps reveals a probability for their occurrence in a good agreement with the simulations. Conversions from three- to fourfold coordinated dopant structures and the subsequent reverse process are significantly less likely than the direct bond inversion. Our results thus provide a model of non-destructive and atomically precise structural modification and detection for two-dimensional materials.