Estimating attribute-based reliability in cognitive diagnostic assessment

  • Author / Creator
    Zhou, Jiawen
  • Cognitive diagnostic assessment (CDA) is a testing format that employs a cognitive model to, first, develop or identify items measuring specific knowledge and skills and, then, use this model to direct psychometric analyses of examinees’ item response patterns to promote diagnostic inferences. The attribute hierarchy method (AHM, Leighton, Gierl, & Hunka, 2004) is a psychometric procedure for classifying examinees’ test item responses into a set of structured attribute patterns associated with different components from a cognitive model of task performance. Attribute reliability is a fundamental concept in cognitive diagnostic assessment because it refers to the consistency of the decisions made in diagnostic test about examinees’ mastery of specific attributes. In this study, an adapted attribute-based reliability estimate was evaluated in comparison of the standard Cronbach’s alpha using simulated data. Factors expected to influence attribute reliability estimates, including test length, sample size, model structure, and model-data fit level, were also studied. Results of this study revealed that the performances of the two attribute-based reliability estimation indices are comparable; however, the adapted index is conceptually more meaningful. Test length, model structure, and model-data fit were shown to impact attribute reliability estimates differentially. Implications to researchers and practitioners were given based on the simulation results. Limitations of the present study and future directions were also discussed.

  • Subjects / Keywords
  • Graduation date
  • Type of Item
  • Degree
    Doctor of Philosophy
  • DOI
  • License
    This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.
  • Language
  • Institution
    University of Alberta
  • Degree level
  • Department
    • Department of Educational Psychology
  • Supervisor / co-supervisor and their department(s)
    • Cui, Ying (Educational Psychology)
    • Gierl, Mark J. (Educational Psychology)
    • Leighton, Jacqueline P. (Educational Psychology)
  • Examining committee members and their departments
    • Carbonaro, Michael (Educational Psychology)
    • Jang, Eunice E. (Education, University of Toronto)
    • Gokiert, Rebecca (Faculty of Extension)