Using the Ottawa Surgical Competency Operative Room Evaluation (O-SCORE) in a Canadian Plastic Surgery Program

• Author / Creator

  Budden, Curtis R

• Competency-based medical education has overtaken the traditional apprenticeship model of medical education. This shift toward competence-based training demands a higher number of assessments per trainee. Current assessment methods in Plastic Surgery have not been well studied. Numerous assessment tools in surgery exist but do not apply to the specialty of Plastic Surgery because of the nature of procedures being assessed. One tool which combines a focus on competency as well as possible applicability to Plastic Surgery is the Ottawa Surgical Competency Operative Room Evaluation (O-SCORE). This tool uses a unique entrustability scale which is construct aligned with competency-based training. The purpose of this study was to examine the validity evidence and reliability of assessments made using the O-SCORE in a Canadian Plastic Surgery training program. Ten residents at the University of Alberta participated in this research and 41 evaluations were performed by supervising Plastic Surgeons. Residents were asked to be evaluated on three common Plastic Surgery procedures (a) breast reduction mammoplasty, (b) mandibular fracture open reduction internal fixation, and (c) and hand fracture fixation. The psychometric properties of the items were assessed using item-total correlation and Cronbach’s alpha. Validity evidence was examined through the differentiation of trainees based on level of training. Reliability testing was performed in two ways. The first was a video analysis of technical skills by Plastic Surgeons. The second was a generalizability study. This G study was a mixed, two facet design. Facets were occasion and item. Occasion in this study was nested within resident as trainees were evaluated on different occasions. The items on the O-SCORE were found to have high item-total correlations (range 0.74 to 0.88). Cronbach’s alpha for technical items based on video analysis was 0.91. It was found that assessments using the O-SCORE significantly differentiated junior and senior trainees on all items and overall score, Λ = 0.08, F(4,36) = 2.77, p < 0.001. The O-SCORE contains a dichotomous item assessing competency and this was found to be significantly different between junior and senior trainees, χ2 = 29.73, ν = 1, p < 0.001. The dependability coefficient from the generalizability study was 0.91. A decision study was conducted and showed that a minimum of two occasions were necessary to obtain a reliability coefficient of greater than 0.85. The O-SCORE and its entrustability-scaled anchors offer a construct-aligned assessment tool for operative skills. At the University of Alberta’s Plastic Surgery residency program, it has been shown that this tool offers high internal consistency of technical items, high overall reliability with three occasions, and validity evidence. Assuring competence is the goal of CMBE and using the O-SCORE as part of an assessment armamentarium is a step closer to this objective.

• Subjects / Keywords

  • Validity
  • Assessement
  • Reliability
  • Surgical education
  • Generalizability theory

• Graduation date

  Fall 2016

• Type of Item

  Thesis

• Degree

  Master of Education

• DOI

  https://doi.org/10.7939/R3CC0TZ8D

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.