Our improving understanding of the life cycle of planetary systems prompts investigations of the role of the Galactic environment before, during and after Asymptotic Giant Branch (AGB) stellar evolution. Here, we investigate the interplay between stellar mass loss, Galactic tidal perturbations, and stellar flybys for evolving stars which host one planet, smaller body or stellar binary companion and reside in the Milky Way's bulge or disc. We find that the potential evolutionary pathways from a main sequence (MS) to a white dwarf (WD) planetary system are a strong function of Galactocentric distance only with respect to the prevalence of stellar flybys. Planetary ejection and collision with the parent star should be more common towards the bulge. At a given location anywhere in the Galaxy, if the mass loss is adiabatic, then the secondary is likely to avoid close flybys during AGB evolution, and cannot eventually escape the resulting WD because of Galactic tides alone. Partly because AGB mass loss will shrink a planetary system's Hill ellipsoid axes by about 20 to 40 per cent, Oort clouds orbiting WDs are likely to be more depleted and dynamically excited than on the MS.