Intelligent Video-based Quality Assessment of Human Activities

• Author / Creator

  Nekoui, Mahdiar

• Can AI-driven robots replace sports officials and rehabilitation physicians in assessing the quality of human activities? An AI judging panel can attend to every nuance of an athlete’s performance and bring more just and agility in scoring. The subjectivity of the judging will be removed and no one will heckle the referee anymore. A trained system can monitor the activities of movement disorder patients and assess their progress in the rehabilitation exercise program. There would no longer be any need for the patients to travel to specialized urban health centers and get feedback on their progress. A computer constantly tracks their movements at home and evaluates their performance.

  The goal of this thesis is to develop some approaches to automatically evaluate the quality of humans' activities from just the video of their performances. The proposed approaches are mostly based on convolutional neural networks that have demonstrated their effectiveness in analyzing visual imagery.

  We first start with grading a diving routine. A human judge keeps track of the coordination among the joints throughout the performance as well as appearance features like the amount of splash and smoothness of the flight. The execution score that the referee provides is then multiplied by the routine's degree of difficulty which is obtained from the official benchmark based on the components of the routine. Inspired by this grading schema, we propose a virtual refereeing system that involves both pose and appearance features of a routine in assessing its execution. On the other hand, a difficulty assessor extracts the components of the flight based on the evolution of the pose throughout the routine. Finally, the overall score is reported by multiplying the difficulty and execution scores.

  We then extend the AI-driven assessor to be applicable to not only other short-term sports like gym-vault and skiing but also minute-long activities like figure-skating. We propose a modular two-stream network that attends to the hierarchical temporal structure of sports routines and can be easily adapted to score minute-long activities. Both fine and coarse-grained temporal dependencies of pose and appearance features are involved in the assessment procedure. In such contortive sports, the athletes usually look for new angles and turns to configure their bodies in some unusual poses. Such pose configurations are not covered by existing pose datasets. Therefore, we introduce a pose-annotated dataset of extreme poses to support experiments in estimating human
  body pose with extreme contortions and involving pose features in action quality assessment (AQA).

  We finally explore the application of AQA in rehabilitation assessment. We develop a self-supervised method that learns the symmetricity and pace of a normal action from off-the-shelf datasets of healthy people activities. These transferable features are then used to assess the activities of a movement disorder patient based on how impaired and slow they are. The proposed method not only demonstrated superior performance in rehabilitation progress evaluation but also showed a good generalization to infants' general movements assessment.

• Subjects / Keywords

  • Computer Vision
  • Machine Learning
  • Video Understanding
  • Action Quality Assessment

• Graduation date

  Spring 2022

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-2x3d-3j18

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.