A Comparison Study of the Reliability Coefficients from Five Approaches to Reliability Estimation

• Author / Creator

  Tang, Wei

• In order to estimate reliability by a single administration of one test form, various approaches and corresponding reliability coefficients have been proposed so far. Currently, the five most influential approaches are: internal consistency, lower bound, principal components analysis (PCA), exploratory factor analysis (EFA), and structural equation modeling (SEM). Facing various approaches and thus dozens of reliability coefficients derived for estimating reliability, practicing researchers are curious to know which reliability coefficient(s) performs best, and under what circumstances. However, a comprehensive comparison of the reliability coefficients from the aforementioned five approaches has not been conducted yet. Therefore, a Monte Carlo study was conducted to evaluate the performances of the reliability coefficients from the five approaches under the conditions that are known to have effect on reliability estimation. Monte Carlo design factors included twelve specific measurement models, two levels of item number, three levels of sample size, three levels of error correlation, and two levels of factor correlation. In total, 72 simulation conditions were created by the combination of all design factors, and each condition was replicated 1,000 times in R environment. The results were collected in two stages. In the first stage, the percentage relative bias, standard error and root mean square error of each reliability coefficient were calculated for each condition. The rounded percentages of estimation failure numbers for each SEM reliability coefficient under all the manipulated conditions were also obtained to identify the conditions with serious estimation issues for the second stage analysis. In the second stage of this study, the percentage relative bias, standard error and root mean square error of Bayesian SEM estimates of reliability for the selected conditions were calculated. Results showed that correctly specified SEM estimates of reliability were least biased and comparatively stable under most of the conditions across the twelve measurement models in this study. However, under the conditions of small item numbers and complicated models, correctly specified SEM estimates of reliability were least accurate and exceptionally unstable due to estimation problems. In addition, over-specified SEM estimates of reliability were examined under the conditions in Model 1 (the tau-equivalent model with independent errors), Model 4 (the congeneric model with independent errors), Model 7 (the correlated factor model with factor correlation at 0.2 and independent errors) and Model 10 (the correlated factor model with factor correlation at 0.6 and independent errors). Results indicated that over-specified SEM estimates of reliability were as accurate and stable as correctly specified SEM estimates of reliability unless estimation problems occurred. Results in the second stage showed that the Bayesian estimation method with non-informative priors could effectively solve estimation problems but fail to eradicate the biases in SEM estimates of reliability. In order to solve estimation problems as well as maintaining the accuracy of SEM estimates of reliability, more types of priors need be tested and compared when using Bayesian estimation methods in a future study.

• Subjects / Keywords

  • Structural-equation-modeling
  • Reliability
  • Bias

• Graduation date

  Fall 2015

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/R3G44HZ6R

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.