Meta-analysis of dependent effect sizes: a multilevel approach

Studies in behavioral sciences are often underpowered, because population effects and sample sizes are rarely impressive. Meta-analyses of multiple studies are therefore highly valuable, because they increase the power. A challenge in most meta-analyses however is that combined effect sizes are dependent, e.g., because a study used multiple criteria, or a research team performed multiple studies. This dependence is often neglected, although this will likely lead to biased results. Up to now, there is no satisfying solution for the problem of dependence.

We will study one promising approach to model dependence: multilevel modeling. Using simulation studies and real data analyses, three-level meta-analyses will be compared to alternative approaches in a variety of scenarios. Existing graphs and meta-analytic two-level techniques will be extended and illustrated. We will also study more complex (but realistic) situations, in which data show a four-level or cross-classified structure. We will answer the questions of minimally required sample sizes at each level to perform these analyses, and how robust conclusions are if simpler models are used. We further will study the advantages of a multivariate approach, rather than using separate univariate analyses for individual regression coefficients, and describe and evaluate a novel Bayesian approach to estimate overall regression and correlation coefficients if studies differ in the predictors that are accounted for.