Challenges in evaluating Welfare to Work policy interventions: would an RCT design have been the answer to all our problems?

We attempted to evaluate the Pathways Advisory Service intervention which aims to provide employment support for Incapacity Benefit recipients, but encountered a number of challenges and barriers to evaluation. This paper explores the issues that arose in designing a suitable evaluation of the Pathways Advisory Service.The main issues that arose were that characteristics of the intervention lead to difficulties in defining a suitable comparison group; and governance restrictions such as uncertainty regarding ethical consent processes and data sharing between agencies for research. Some of these challenges threatened fundamentally to limit the validity of any experimental or quasi-experimental evaluation we could design - restricting recruitment, data collection and identification of an appropriate comparison group. Although a cluster randomised controlled trial design was ethically justified to evaluate the Pathways Advisory Service, this was not possible because the intervention was already being widely implemented. However, this would not have solved other barriers to evaluation. There is no obvious method to perform a controlled evaluation for interventions where only a small proportion of those eligible are exposed. Improved communication between policymakers and researchers, clarification of data sharing protocols and improved guidelines for ethics committees are tangible ways which may reduce the current obstacles to this and other similar evaluations of policy interventions which tackle key determinants of health.The evaluation of social interventions is hampered by more than their suitability to randomisation. Data sharing, participant identification and recruitment problems are common to randomised and non-randomised evaluation designs. These issues require further attention if we are to learn from current social policy.In 1999 the Acheson report highlighted the lack of evidence around interventions to reduce health inequalities, and recommended that all po