Development and Validation of an Evaluation Capacity Instrument for the Early Childhood Field

• Author / Creator

  El Hassar, Btissam

• This study contributes to the conceptualization and measurement of evaluation capacity (EC) by developing and validating an EC instrument and an empirically based conceptual EC framework that is relevant to the context of the early childhood field in Alberta, Canada. As evaluation has become a crucial strategy for addressing learning and accountability demands faced by organizations, a common challenge for organizations is how to access adequate capacity for conducting evaluations and meaningfully using the results. Previous research has defined and conceptualized the construct of EC, but less attention has been given to its measurement, especially measurement that is grounded in context. This gap warranted a new examination to assess the factors that contribute to building effective EC within specific contexts, the relationships among the factors, and the expected outcomes of EC efforts. The need for better measurement of EC is evidenced by numerous calls from scholars and practitioners as recently as 2017. A cross-sectional survey design was used to answer the study’s research questions about the underlying factor structure of the EC construct, the empirical relationships among these factors, and the influence of participants’ organizational roles on the EC relationships. The study’s research approach conforms to the contemporary view of validity, in which multiple types of evidence are produced during the validation process to substantiate the validity argument. The research began by drawing on the literature and consulting experts who are knowledgeable about the study’s context to identify and operationalize relevant EC components and develop an EC instrument that is empirically sound and contextually relevant. Data were then collected from participants in the early childhood field using the instrument. Exploratory and confirmatory factor analyses were used to evaluate measurement models that were based on individual and organizational EC scales. These results suggest that both individual and organizational EC constructs are multidimensional and that they are adequately measured by the developed EC instrument. Structural equation modelling was then used to evaluate direct and indirect relationships among the individual and organizational factors that were hypothesized within the EC framework. The direct and indirect relationships were found to be statistically significant. Specifically, the latent variables Organizational Culture, Organizational Leadership, Organizational Commitment to Evaluation, and Individual Evaluation Skills, and Individual Attitude Toward Evaluation positively predicted Individual Motivation to Conduct and Use Evaluations. Furthermore, these relationships explained 49% of the variance in Individual Motivation. Subsequently, multi-group path analysis was used to assess the influence of respondents’ roles within their organizations on these relationships. The multi-group path analysis results revealed that respondents’ professional roles did not statistically influence the direct and indirect EC relationships.This study offers important implications for theory, methodology, and practice. EC theory is advanced through the contribution of an empirical EC framework specifying the magnitude and directionality of the influences among individual and organizational EC factors. In so doing, the framework highlights the interrelatedness of the factors and thus the importance of adopting a systems perspective to inform EC conceptualization, measurement and initiatives. A context-relevant methodology contributes to generating valid conclusions about the extent to which EC exists within individuals and organizations. In so doing, the methodology provides an illustrative example for scholars to apply the study procedures in contexts beyond the early childhood field. Finally, the instrument enhances the capacity for practitioners to access data that may inform EC building initiatives. In this way, the instrument can be used by individuals and organizations for evidence-based decision-making that targets specific EC needs.

• Subjects / Keywords

  • Conceptualization of Evaluation Capacity
  • Structural Equation Model
  • Evaluation Capacity
  • Survey Design
  • Measuring Evaluation Capacity
  • Validation Study
  • Confirmatory Factor Analysis

• Graduation date

  Spring 2019

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/r3-0r2y-ab32

• License

  Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.