Estimating attribute-based reliability in cognitive diagnostic assessment

• Author / Creator

  Zhou, Jiawen

• Cognitive diagnostic assessment (CDA) is a testing format that employs a
  cognitive model to, first, develop or identify items measuring specific knowledge
  and skills and, then, use this model to direct psychometric analyses of
  examinees’ item response patterns to promote diagnostic inferences. The
  attribute hierarchy method (AHM, Leighton, Gierl, & Hunka, 2004) is a
  psychometric procedure for classifying examinees’ test item responses into a set
  of structured attribute patterns associated with different components from a
  cognitive model of task performance. Attribute reliability is a fundamental
  concept in cognitive diagnostic assessment because it refers to the consistency
  of the decisions made in diagnostic test about examinees’ mastery of specific
  attributes. In this study, an adapted attribute-based reliability estimate was
  evaluated in comparison of the standard Cronbach’s alpha using simulated data.
  Factors expected to influence attribute reliability estimates, including test length,
  sample size, model structure, and model-data fit level, were also studied.
  Results of this study revealed that the performances of the two attribute-based
  reliability estimation indices are comparable; however, the adapted index is
  conceptually more meaningful. Test length, model structure, and model-data fit
  were shown to impact attribute reliability estimates differentially. Implications
  to researchers and practitioners were given based on the simulation results.
  Limitations of the present study and future directions were also discussed.

• Subjects / Keywords

  • Reliability, cognitive diagnostic assessment

• Graduation date

  Spring 2010

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/R3C712

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.