My colleagues and I published an article “Calculating power for multilevel implementation trials in mental health: Meaningful effect sizes, intraclass correlation coefficients, and proportions of variance explained by covariates” in the Journal of Implementation Research and Practice.
Here’s a quick overview of some important points
TL;DR: Designing implementation studies in mental health settings? Make sure to understand the statistical parameters critical for power analysis, especially effect sizes, ICC, and covariate R².
Effect Sizes, ICCs, and Covariate R²s Are Crucial for Multilevel Trials:
- In multilevel studies (e.g., patients nested within providers, providers within clinics), it’s crucial to account for the clustering effects when determining sample sizes. Small changes in these parameters can dramatically affect the number of sites or participants needed for robust results.
- Effect sizes reflect the expected difference in outcomes between intervention groups. The intraclass correlation coefficient (ICC) tells us how similar individuals are within the same cluster, while covariate R² explains how much variance in the outcome is due to baseline characteristics.
We Collected Data from Trials to Provide Reference Values:
- We reviewed mental health-focused implementation trials funded by NIMH from 2010-2020.
- From these trials, we extracted 53 implementation and clinical outcome variables, calculating effect sizes, ICCs, and covariate R²s. This data helps future researchers plan studies more accurately by offering empirical reference values for power calculations.
Effect Size and ICC Vary by Trial Design:
- The effect size (Cohen’s d) for standard vs. enhanced implementation conditions ranged widely across studies.
- ICC values were divided into three categories based on study designs: repeated measures (time within person), cross-sectional level 2 (e.g., participants within clusters), and cross-sectional level 3 (e.g., participants within higher-level clusters). These categories help researchers select appropriate ICCs for their designs.
Outcome Measurement Matters:
- Whether outcomes are measured via self-report or external observation (e.g., rater-coded) can significantly affect the ICC and, consequently, the power and precision of the study.
Practical Application:
- This study’s findings help ensure better planning and execution of future implementation research in mental health. Having accurate estimates of effect sizes, ICC, and covariate R²s allows for more efficient and impactful studies, ultimately contributing to the advancement of evidence-based mental health interventions.
By understanding these metrics, researchers can optimize study designs and use scientific resources more effectively.
References:
- Hedges LV, Hedberg EC. Intraclass Correlation Values for Planning Group-Randomized Trials in Education. Educational Evaluation and Policy Analysis. 2007;29:60–87.
- Snijders TAB, Bosker RJ. Multilevel analysis: an introduction to basic and advanced multilevel modeling. London ; Thousand Oaks, Calif: Sage Publications; 1999.
- Proctor, E., Silmere, Ha Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and policy in mental health and mental health services research, 38, 65-76.
- Zeileis, A., Lumley, T., Berger, S., Graham, N., & Zeileis, M. A. (2019). Package ‘sandwich’. R package version, 2-5.
Link to to the publication at the Journal of Implementation Research and Practice.