Evaluating the Performance of the Uncorrected and Corrected Reliability Alpha for Range Restriction and the Confidence Intervals in a Single and Meta-Analytic Study

• Author / Creator

  Li, Johnson C. H.

• Range restriction has long been a methodological problem in educational and psychological research (Hunter & Schmidt, 2004), and this usually leads to a downward-biased estimate of a statistic. Even though much research has examined the performance of Pearson’s correlation under range restriction in both single and meta-analytic studies (e.g., Li, Chan, & Cui, 2011a), the assessment of reliability coefficients (e.g., coefficient alpha) under range restriction is relatively limited. Regarding a single study, Fife, Mendoza, and Terry’s (2012) have recently examined the performance of the uncorrected and bias-corrected coefficient alpha; as an extension, the performance of the confidence intervals (CIs) and widths also need to be examined. Regarding a meta-analytic study, Rodriguez and Maeda (2006) have proposed a framework for conducting a meta-analysis of coefficient alpha; as an extension, the accuracy of the bias-corrected mean alpha as well as the associated CIs also need to be evaluated.
  In light of these unexamined issues, this dissertation sought to evaluate the performance of the uncorrected and bias-corrected alphas—as well as their CI—in both single and meta-analytic study research situations. This provides a comprehensive assessment of reliability under range restriction, thereby providing guidelines about the treatment of biases that come from range restriction. The Monte Carlo results showed that the uncorrected alpha suffered as a function of the selection ratio and the correlation between the test and the selection variable in both single and meta-analytic studies. By contrast, the bias-corrected alpha could adjust for the bias appropriately. Moreover, the bootstrap CIs constructed for the bias-adjusted alpha in both single and meta-analytic studies were generally accurate across different simulation conditions, including sample size, item number, etc. Application of the correction procedure and CI construction in a real study is demonstrated. Based on these results, conclusions, discussions, and recommendations are also presented.

• Subjects / Keywords

  • Range restriction
  • Meta-analysis
  • Reliability

• Graduation date

  Spring 2013

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/R3CD4X

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.