Climate change has resulted in increased vulnerability of smallholder farmers in marginal areas of Zimbabwe where there is limited capacity to adapt to changing climate. One approach that has been used to adapt to changing climate is in-field water harvesting for improved crop yields in the semi- arid regions of Zimbabwe. This review analyses the history of soil and water conservation in Zimbabwe, efforts of improving water harvesting in the post independence era, farmer driven innovations, water harvesting technologies from other regions, and future directions of water harvesting in semi arid marginal areas. From this review it was observed that the blanket recommendations that were made on the early conservation method were not suitable for marginal areas as they resulted in increased losses of the much needed water. In the late 1960 and 70s', soil and water conservation efforts was a victim of the political environment and this resulted in poor uptake. Most of the water harvesting innovations which were promoted in the 1990s' and some farmer driven innovations improved crop yields in marginal areas but were poorly taken up by farmers because they are labour intensive as the structures should be made annually. To address the challenges of labour shortages, the use of permanent in-field water harvesting technologies are an option. There is also need to identify ways for promoting water harvesting techniques that have been proven to work and to explore farmer-led knowledge sharing platforms for scaling up proven technologies.