An empirical study using permutation-based resampling in meta-regression

We isolated a sample of randomized controlled clinical trials (RCTs) for interventions that have a small number of trials (herbal medicine trials). Trials were then grouped by herbal species and condition and assessed for methodological quality using the Jadad scale, and data were extracted for each outcome. Finally, we performed meta-analyses on the primary outcome of each group of trials and meta-regression for methodological quality subgroups within each meta-analysis. We used large sample methods and permutation methods in our meta-regression modeling. We then compared final models and final P values between methods.We collected 110 trials across 5 intervention/outcome pairings and 5 to 10 trials per covariate. When applying large sample methods and permutation-based methods in our backwards stepwise regression the covariates in the final models were identical in all cases. The P values for the covariates in the final model were larger in 78% (7/9) of the cases for permutation and identical for 22% (2/9) of the cases.We present empirical evidence that permutation-based resampling may not change final models when using backwards stepwise regression, but may increase P values in meta-regression of multiple covariates for relatively small amount of trials.Systematic reviews are prone to various forms of heterogeneity between included studies. Variability in the participants, interventions and outcomes across studies may be termed clinical heterogeneity; variability in the trial design and quality is typically termed methodological heterogeneity; variability in treatment effects between trials can be termed statistical heterogeneity [1,2]. Methodological heterogeneity hinges on the exact methods of the individual trials, and how they differ from each other. That is, trials that do not properly conceal allocation to treatment groups may bias estimates in treatment effect and cause increased variations in effect between studies included systematic reviews [3]. Signifi