Download the full-sized PDF of Methods for determining whether subscore reporting is warranted in large-scale achievement assessmentsDownload the full-sized PDF



Permanent link (DOI):


Export to: EndNote  |  Zotero  |  Mendeley


This file is in the following communities:

Graduate Studies and Research, Faculty of


This file is in the following collections:

Theses and Dissertations

Methods for determining whether subscore reporting is warranted in large-scale achievement assessments Open Access


Other title
large-scale assessment of student achievement
subscore reporting
Type of item
Degree grantor
University of Alberta
Author or creator
Babenko, Oksana Illivna
Supervisor and department
Cui, Ying (Educational Psychology)
Rogers, W. Todd (Educational Psychology)
Examining committee member and department
Anderson, John (Educational Psychology and Leadership Studies)
Mrazik, Martin (Educational Psychology)
Norris, Stephen (Educational Policy Studies)
Parrila, Rauno (Educational Psychology)
Department of Educational Psychology

Date accepted
Graduation date
Doctor of Philosophy
Degree level
Officials of large-scale assessment programs often want to report subscale scores in addition to the total test score. However, in addition to the reliability of reported scores, evidence that subscales reveal real differences in student performances must be obtained in order to support reporting of subscale scores. In this study, two correlational methods, including correlations corrected for attenuation, r’, and the proportional reduction of the mean squared error, PRMSE (Haberman, 2005; Sinharay et al., 2007), and the agreement method (Kelley, 1923) for determining whether subscore reporting is warranted in large-scale achievement assessments were examined. Whereas correlation-based methods consider student performances on pairs of measures in terms of ranked positions, the agreement method takes into account actual differences between students’ standard scores on the pairs of measures being compared. The correlational methods revealed that with one possible subscale difference, the subscales did not differ among themselves and from the total test for the English Reading (N = 128,089) and Mathematics (N = 127,596) assessments considered in this study. In contrast, Kelley’s agreement method one to five percent students had differences between their scores on the English Reading subscales that were greater than the difference expected due to the chance. However, with two exceptions for the Mathematics assessment, the results of the agreement method were uninterpretable. In agreement with Sinharay, et al. (2007), it was concluded that for the detection methods to work, three conditions need to be met, one substantive (multidimensional construct for which scores are wanted for each dimension), and two statistical (high reliabilities of and low intercorrelations among subscales). The results for replicated random samples (n = 250, 500, 1,000, 2,000, and 5,000) revealed that the statistics for the three detection methods were accurate and precise estimators of the corresponding population parameters.
Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.
Citation for previous publication

File Details

Date Uploaded
Date Modified
Audit Status
Audits have not yet been run on this file.
File format: pdf (Portable Document Format)
Mime type: application/pdf
File size: 560019
Last modified: 2015:10:12 13:17:44-06:00
Filename: Babenko_Oksana_Fall 2011.pdf
Original checksum: a8ae36b85a09e91698c3c189e51b7d9a
Well formed: false
Valid: false
Status message: Invalid page tree node offset=322446
Status message: Unexpected error in findFonts java.lang.ClassCastException: edu.harvard.hul.ois.jhove.module.pdf.PdfSimpleObject cannot be cast to edu.harvard.hul.ois.jhove.module.pdf.PdfDictionary offset=2553
Status message: Invalid Annotation list offset=463653
Status message: Outlines contain recursive references.
File title: Babenko, Oksana_Prefatory pages_Sept 27, 2011
Activity of users you follow
User Activity Date