A Simple, Principled Approach to Combining Evidence From Meta

Recent discussions of the influence of publication bias and questionable research practices on psychological science have increased researchers’ interest in both bias-correcting meta-analytic techniques and preregistered replication. Both approaches have their strengths: For example, meta-analyses can quantitatively characterize the full body of work done in the field of interest, and preregistered replications can be immune to bias. Both approaches also have clear weaknesses: Decisions about which meta-analytic estimates to interpret tend to be controversial, and replications can be discounted for failing to address important methodological heterogeneity. Using the experimental literature on ego depletion as a case study, we illustrate a principled approach to combining information from meta-analysis with information from subsequently conducted high-quality replications. This approach (a) compels researchers to explicate their beliefs in meta-analytic conclusions (and also, when controversy arises, to defend the basis for those beliefs), (b) encourages consideration of practical significance, and (c) facilitates the process of planning replications by specifying the sample sizes necessary to have a reasonable chance of changing the minds of other researchers