ERA

Download the full-sized PDF of A Comparison Study of the Reliability Coefficients from Five Approaches to Reliability EstimationDownload the full-sized PDF

Analytics

Share

Permanent link (DOI): https://doi.org/10.7939/R3G44HZ6R

Download

Export to: EndNote  |  Zotero  |  Mendeley

Communities

This file is in the following communities:

Graduate Studies and Research, Faculty of

Collections

This file is in the following collections:

Theses and Dissertations

A Comparison Study of the Reliability Coefficients from Five Approaches to Reliability Estimation Open Access

Descriptions

Other title
Subject/Keyword
structural equation modeling
reliability
bias
Type of item
Thesis
Degree grantor
University of Alberta
Author or creator
Tang, Wei
Supervisor and department
Cui, Ying (Department of Educational Psychology)
Examining committee member and department
Gokiert, Rebecca (The Faculty of Extension)
Gierl, Mark (Department of Educational Psychology)
Cormier, Damien (Department of Educational Psychology)
Roberts, Mary Roduta (Occupational Therapy)
Department
Department of Educational Psychology
Specialization
Measurement, Evaluation, and Cognition
Date accepted
2015-06-29T13:57:34Z
Graduation date
2015-11
Degree
Doctor of Philosophy
Degree level
Doctoral
Abstract
In order to estimate reliability by a single administration of one test form, various approaches and corresponding reliability coefficients have been proposed so far. Currently, the five most influential approaches are: internal consistency, lower bound, principal components analysis (PCA), exploratory factor analysis (EFA), and structural equation modeling (SEM). Facing various approaches and thus dozens of reliability coefficients derived for estimating reliability, practicing researchers are curious to know which reliability coefficient(s) performs best, and under what circumstances. However, a comprehensive comparison of the reliability coefficients from the aforementioned five approaches has not been conducted yet. Therefore, a Monte Carlo study was conducted to evaluate the performances of the reliability coefficients from the five approaches under the conditions that are known to have effect on reliability estimation. Monte Carlo design factors included twelve specific measurement models, two levels of item number, three levels of sample size, three levels of error correlation, and two levels of factor correlation. In total, 72 simulation conditions were created by the combination of all design factors, and each condition was replicated 1,000 times in R environment. The results were collected in two stages. In the first stage, the percentage relative bias, standard error and root mean square error of each reliability coefficient were calculated for each condition. The rounded percentages of estimation failure numbers for each SEM reliability coefficient under all the manipulated conditions were also obtained to identify the conditions with serious estimation issues for the second stage analysis. In the second stage of this study, the percentage relative bias, standard error and root mean square error of Bayesian SEM estimates of reliability for the selected conditions were calculated. Results showed that correctly specified SEM estimates of reliability were least biased and comparatively stable under most of the conditions across the twelve measurement models in this study. However, under the conditions of small item numbers and complicated models, correctly specified SEM estimates of reliability were least accurate and exceptionally unstable due to estimation problems. In addition, over-specified SEM estimates of reliability were examined under the conditions in Model 1 (the tau-equivalent model with independent errors), Model 4 (the congeneric model with independent errors), Model 7 (the correlated factor model with factor correlation at 0.2 and independent errors) and Model 10 (the correlated factor model with factor correlation at 0.6 and independent errors). Results indicated that over-specified SEM estimates of reliability were as accurate and stable as correctly specified SEM estimates of reliability unless estimation problems occurred. Results in the second stage showed that the Bayesian estimation method with non-informative priors could effectively solve estimation problems but fail to eradicate the biases in SEM estimates of reliability. In order to solve estimation problems as well as maintaining the accuracy of SEM estimates of reliability, more types of priors need be tested and compared when using Bayesian estimation methods in a future study.
Language
English
DOI
doi:10.7939/R3G44HZ6R
Rights
Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.
Citation for previous publication

File Details

Date Uploaded
Date Modified
2015-06-29T19:57:36.839+00:00
Audit Status
Audits have not yet been run on this file.
Characterization
File format: pdf (PDF/A)
Mime type: application/pdf
File size: 4523573
Last modified: 2016:06:24 17:40:18-06:00
Filename: Tang_Wei_201506_PhD.pdf
Original checksum: cbfdd99c0e63d5fbbad41dbc164abc4f
Activity of users you follow
User Activity Date