2 Citations and Resources

2.1 R Code Online Examples

Shaw, M. and Flake J.K. (2022) Introduction to Multilevel Modelling

2.2 MLM: Multilevel Modeling, aka Mixed Effects Regression

2.2.1 Major Textbooks

Fitzmaurice, G. M., Laird, N. M., Ware, J. H. (2004). Applied Longitudinal Analysis. Hoboken, NJ: John Wiley & Sons.

Hox, J. J., Moerbeek, M., & Van de Schoot, R. (2017). Multilevel analysis: Techniques and applications. Routledge.

Gelman, A., & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge University Press.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical Linear Models: Applications and Data Analysis Methods (Vol. 1). Sage.

Singer, J. D., Willett, J. B., & Willett, J. B. (2003). Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. Oxford University Press.

Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Los Angeles; London, SAGE.

2.2.2 Articles

Nezlek, J. B. (2012). Multilevel modeling for psychologists.

Nezlek, J. B. (2020): Applications of multilevel modeling in psychological science: Intensive repeated measures designs.

Huang, F. L. (2018). Multilevel modeling myths. School Psychology Quarterly, 33(3), 492.

Boisgontier, M. P., & Cheval, B. (2016). The anova to mixed model transition. Neuroscience & Biobehavioral Reviews, 68, 1004-1005.

Quené, H., & Van den Bergh, H. (2004). On multi-level modeling of data from repeated measures designs: A tutorial. Speech communication, 43(1-2), 103-121.

2.2.3 Websites

Interactive web-app: [An Introduction to Hierarchical Modeling}(http://mfviz.com/hierarchical-models/) (Michael Freeman)
Blog post: https://geekflare.com/multilevel-modeling/
The analysis Factor: Five Advantages of Running Repeated Measures ANOVA as a Mixed Model

2.2.4 YouTube

John Nezlek talk: An Introduction to Multilevel Modeling - basic terms and research examples This video was also optional in the first week. I know it is long (1 hr 45 min) but its SOOOOOOO good. If nothing more, watch the first 40 minutes. It covers the multilevel philosophy and lots of simple illustrative examples, including the paradox/confounding of not doing MLM when your data is clustered.
- at 42-48 the ICC issue
- at 48-52 power and sample size
- at 52-1:11 the logic of multilevel models (slopes as outcomes)
- at 1:19 he starts going into longitudinal (don’t watch yet)
- at 1:30-1:34 should you use MLM
- at 1:35-1:38 dyatic data (partners) but, warning, he does swear some
Thomas Grund: What is an ecological fallacy? (15 min)
Dr. Trefor Bazett: How SIMPSON’s PARADOX explains weird COVID 19 statistics (8 min)
Mikko Ronkko: Introduction to Multilevel Data (7 min) <- ecological fallacy & Simpson’s Paradox
Violet Brown: Multi-Level Modeling, Part 1 (21 min)
Quant Psych: Mixed Model Notation - A Simple Explanation (20 min) (longitudinal example)
Ian Brunton-Smith: Multilevel Models: Random Intercept Models (8.5 min)
Ian Brunton-Smith: Multilevel Models: Random Coefficients Models (8.5 min)
Violet Brown: Multi-level Modeling, Part 2 (26 min)

2.3 Fixed vs Random Effects

2.4 Centering Predictor Variables: Grand Mean vs Group Means

Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: a new look at an old issue. Psychological methods, 12(2), 121.

2.4.1 Articles

Enders, C.K. (2013) Centering Predictors and Contextual Effects. The SAGE Handbook of Multilevel Modeling, 89-108. DOI:https://doi.org/10.4135/9781446247600.n6

Bliese, P. D., Maltarich, M. A., & Hendricks, J. L. (2018). Back to basics with mixed-effects models: Nine take-away points. Journal of Business and Psychology, 33(1), 1-23.

Hamaker, E. L., & Muthén, B. (2020). The fixed versus random effects debate and how it relates to centering in multilevel modeling. Psychological methods, 25(3), 365.

Paccagnella, O. (2006). Centering or not centering in multilevel models? The role of the group mean and the assessment of group effects. Evaluation review, 30(1), 66-85.

Bell, A., Jones, K., & Fairbrother, M. (2018). Understanding and misunderstanding group mean centering: A commentary on Kelley et al.’s dangerous practice. Quality & quantity, 52(5), 2031-2036.

2.4.2 Websites

The Analysis Factor: When NOT to Center a Predictor Variable in Regression
MetricGate (2026) Group-Mean vs Grand-Mean Centering in Multilevel Models <– INTERACTIVE!

2.4.3 YouTube

Mikko Ronkko: Grand-mean centering, cluster-mean centering, and cluster means (13 min)
Christian Geiser (QuantFish): Why You Should Center Variables in Statistics (11 min, last 2 min)

2.5 Effect Size: Variance Explained (R-squared like measures)

Hox, J. J. (2017). Multilevel analysis: techniques and applications (3rd ed). New York: Routledge.

Johnson, P. C. D. (2014). Extension of Nakagawa and Schielzeth’s R2 GLMM to random slopes models. Methods in Ecology and Evolution, 5(9), 944–946. doi:10.1111/2041-210X.12225

Nakagawa, S., and Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. doi:10.1111/j.2041-210x.2012.00261.x

Nakagawa, S., Johnson, P. C. D., and Schielzeth, H. (2017). The coefficient of determination R2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. Journal of The Royal Society Interface, 14(134), 20170213.

2.5.1 Articles

Lorah, J. (2018). Effect size measures for multilevel models: Definition, interpretation, and TIMSS example. Large-scale assessments in education, 6(1), 8.

Pek, J., & Flora, D. B. (2018). Reporting effect sizes in original psychological research: A discussion and tutorial. Psychological methods, 23(2), 208.

Rights, J. D., & Cole, D. A. (2018). Effect size measures for multilevel models in clinical child and adolescent research: New R-squared methods and recommendations. Journal of Clinical Child & Adolescent Psychology, 47(6), 863-873.

Lai, M. H., & Kwok, O. M. (2016). Estimating standardized effect sizes for two-and three-level partially nested data. Multivariate Behavioral Research, 51(6), 740-756.

Grob, J., & Moller, A. (2025). Effect Size Estimation in Linear Mixed Models. Metron. 83(3), 353-363.

2.5.2 YouTube

Quant Psych: Explaining Variance Explained in Mixed Models (10 min)

2.6 Effect Size: Pairwise Differences (Standardized Mean Difference, SMD)

There are a number of ways to calculate standardized mean difference (SMD) effect sizes and there really is not a universally accepted “right” way. Using residual SD and the square root of the sum of variance components from a mixed model for standardization are NOT traditional Cohen’s d statistics, so they should be referred to as standardized effect sizes.

The important thing is to detail which method you are using and make sure to label it appropriately.

Probably the effect size calculation with the most academic support is taking the square root of the sum of all variance components in the model. This method was first discussed by Westfall, Judd, and Kenny (2014) and later included in a statistical tutorial on Effect Sizes and Power Analysis in Mixed Effects Models by Marc Brysbaert and Michael Stevens (2018).

We can add interpretations to these SMD effects. There have been different propositions regarding the qualitative interpretation of effect sizes’. The original interpretation given by Cohen suggests:

Small: < 0.5
Moderate: 0.5 - 0.7
Large: > 0.8

More recently Will G. Hopkins suggested a revised scale with threshold values at:

0.2 small
0.6 moderate
1.2 large
2.0 very large
4.0 nearly perfect

Brysbaert, M., & Stevens, M. (2018). Power analysis and effect size in mixed effects models: A tutorial. Journal of cognition, 1(1).

Luke, S. G. (2017). Evaluating significance in linear mixed-effects models in R. Behavior research methods, 49(4), 1494-1502.

Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143(5), 2020.

2.6.1 Example: Write-up for Methods-Analysis section

Follow-up pairwise comparisons utilizing Kenward-Rodger degrees of freedom (Luke, 2017) were made without/with?? controlling for multiple comparisons. Corresponding effect sizes give Cohen’s d-like standardized mean differences (SMD) utilizing the standard deviation based on the sum of the MLM variance components (Brysbaert & Stevens, 2018).

2.7 GLMM: Generalized Linear Multilevel Models or Generalized Mixed Effects Regression

2.8 GEE: Generalized Estimating Equations

Fitzmaurice, G. M., Laird, N. M., Ware, J. H. (2004). Applied Longitudinal Analysis. Hoboken, NJ: John Wiley & Sons.

Hardin, J. W., Hilbe, J. M. (2003). Generalized Estimating Equations. Boca Raton: Chapman & Hall/CRC.

Liang, K.-Y., Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika 73:13–22.

McCullagh, P., Nelder, J. A. (1989). Generalized Linear Models. 2nd ed. London: Chapman & Hall.

Nelder, J. A., Wedderburn, R. W. M. (1972). Generalized linear models. J. Roy. Statist. Soc.,Ser. A 135:370–384.

Burger, H. G., Dudley, E. C., Cui, J., Dennerstein, L., Hopper, J. L. (2000). A prospective longitudinal study of serum testosterone levels during and after the menopause transition. J. Clin. Endocrinol. Metabolism 85:2832–2838.

2.8.1 QIC: Comparing GEE Models

The QIC involves using the quasi-likelihood constructed under the working independence model and the naive and robust covariance estimates of estimated regression coefficients.

QIC-U is asymptotic, not appropriate for selected an optimal correlation structure

Cui, J., & Qian, G. (2007). Selection of working correlation structure and best model in GEE analyses of longitudinal data. Communications in statistics—Simulation and computation®, 36(5), 987-996. Download PDF

We propose to choose the best correlation structure first and then choose the best subsets of covariates. The best correlation structure was selected based on the full model including all covariates. However, if we use only a subset of covariates to select the best correlation structure, we may end up with a different correlation structure. This is a dilemma as to choose the best correlation structure first or the best subset of covariates first. It seems that the full model contains most information to predict the outcome variable and is reasonable to use it to select the best correlation structure first (Hardin and Hilbe, 2003).

Burnham, K. P. and D. R. Anderson. 2002. Model selection and multimodel inference: a practical information-theoretic approach. Second edition. Springer Science and Business Media, Inc., New York. Download PDF

Pan, W. 2001. Akaike’s information criterion in generalized estimating equations. Biometrics 57:120-125. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.0006-341X.2001.00120.x Download PDF

Wedderburn, R. W. M. (1974). Quasi-likelihood functions, generalized liner models, and the Gauss-newton method. Biometrika 61:439–447.