In order to monitor the changes of the glaciers in the Gongga Mountain region on the south-eastern margin of the Qinghai-Tibetan Plateau, 74 monsoonal temperate glaciers were investigated by comparing the Chinese Glacier Inventory (CGI), recorded in the 1960s, with Landsat MSS in 1974, Landsat TM in 1989, 1994, 2005, and ASTER data in 2009. The remote sensing data have been applied to map the glacier outline by threshold ratio images (TM4/TM5). Moreover, the glacier outlines were verified by GPS survey on four large glaciers (Hailuogou (HLG), Mozigou (MZG), Yanzigou (YZG), and Dagongba (DGB)) in 2009. The results show that the area dominated by the 74 glaciers has shrunk by 11.3% (29.2 km2) from 1966 to 2009. Glacier area on the eastern and western slopes of the Gongga Mountains decreased by 9.8% and 14.6% since 1966, respectively. The loss in glacier area and length is, respectively, 0.8 km2 and 1146.4 m for the HLG Glacier, 2.1 km2 and 501.8 m for the MZG Glacier, 0.8 km2 and 724.8 m for the YZG Glacier, and 2.4 km2 and 1002.3 m for the DGB Glacier. Decades of climate records obtained from three meteorological stations in the Gongga Mountains were analyzed to evaluate the impact of the temperature and precipitation on glacier retreat. The mean annual temperatures over the eastern and western slopes of the Gongga Mountains have been increasing by 0.34 K decade 1 and 0.24 K decade 1 (1988–2009), respectively. Moreover, mean annual precipitation has only increased by 1% in the past 50 yr. The increasing amount of precipitation could not compensate for the glacier mass loss due to the temperature increase in the Gongga Mountains. This suggests that the warming of the climate is probably also responsible for the glacier retreat in the study region. At the region scale, glacier changes were also controlled by local topographical factors.