Download the full-sized PDF of Novel Methods for Robust Real-time Hand Gesture InterfacesDownload the full-sized PDF



Permanent link (DOI):


Export to: EndNote  |  Zotero  |  Mendeley


This file is in the following communities:

Graduate Studies and Research, Faculty of


This file is in the following collections:

Theses and Dissertations

Novel Methods for Robust Real-time Hand Gesture Interfaces Open Access


Other title
Hand pose recognition
Computer Vision
Gesture recognition
Type of item
Degree grantor
University of Alberta
Author or creator
Rossol, Nathaniel S.
Supervisor and department
Irene Cheng (Computing Science)
Anup Basu (Computing Science)
Examining committee member and department
Matthew Turk (Computer Science)
Nilanjan Ray (Computing Science)
Witold Pedrycz (Electrical and Computer Engineering)
Anup Basu (Computing Science)
Irene Cheng (Computing Science)
Department of Computing Science

Date accepted
Graduation date
Doctor of Philosophy
Degree level
Real-time control of visual display systems via mid-air hand gestures offers many advantages over traditional interaction modalities. In medicine, for example, it allows a practitioner to adjust display values, e.g. contrast or zoom, on a medical visualization interface without the need to re-sterilize the interface. However, there are many practical challenges that make such interfaces non-robust including poor tracking due to frequent occlusion of fingers, interference from hand-held objects, and complex interfaces that are difficult for users to learn to use efficiently. In this work, various techniques are explored for improving the robustness of computer interfaces that use hand gestures. This work is focused predominately on real-time markerless Computer Vision (CV) based tracking methods with an emphasis on systems with high sampling rates. First, we explore a novel approach to increase hand pose estimation accuracy from multiple sensors at high sampling rates in real-time. This approach is achieved through an intelligent analysis of pose estimations from multiple sensors in a way that is highly scalable because raw image data is not transmitted between devices. Experimental results demonstrate that our proposed technique significantly improves the pose estimation accuracy while still maintaining the ability to capture individual hand poses at over 120 frames per second. Next, we explore techniques for improving pose estimation for the purposes of gesture recognition in situations where only a single sensor is used at high sampling rates without image data. In this situation, we demonstrate an approach where a combination of kinematic constraints and computed heuristics are used to estimate occluded keypoints to produce a partial pose estimation of a user's hand which is then used with our gestures recognition system to control a display. The results of our user study demonstrate that the proposed algorithm significantly improves the gesture recognition rate of the setup. We then explore gesture interface designs for situations where the user may (or may not) have a large portion of their hand occluded by a hand-held tool while gesturing. We address this challenge by developing a novel interface that uses a single set of gestures designed to be equally effective for fingers and hand-held tools without the need for any markers. The effectiveness of our approach is validated through a user study on a group of people given the task of adjusting parameters on a medical image display. Finally, we examine improving the efficiency of training for our interfaces by automatically assessing key user performance metrics (such as dexterity and confidence), and adapting the interface accordingly to reduce user frustration. We achieve this through a framework that uses Bayesian networks to estimate values for abstract hidden variables in our user model, based on analysis of data recorded from the user during operation of our system.
This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for the purpose of private, scholarly or scientific research. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.
Citation for previous publication
N. Rossol, I. Cheng, I. Jamal, R. Shen, and A. Basu. Touchfree medical interfaces, in Engineering in Medicine and Biology Society (EMBC), 2014 36th Annual International Conference of the IEEE, Aug 2014, pp. 6597--6600.N. Rossol, I. Cheng, W. F. Bischof, and A. Basu. A framework for adaptive training and games in virtual reality rehabilitation environments. In Proceedings of the 10th International Conference on Virtual Reality Continuum and Its Applications in Industry, VRCAI'11, pages 343-346, New York, NY, USA, 2011. ACM.N. Rossol, I. Cheng, and A. Basu. A multi-sensor technique for hand pose estimation through novel skeletal pose analysis. IEEE Transactions on Human Machine Systems, 2015. (Submitted and under revision).

File Details

Date Uploaded
Date Modified
Audit Status
Audits have not yet been run on this file.
File format: pdf (PDF/A)
Mime type: application/pdf
File size: 7312183
Last modified: 2016:09:01 15:57:48-06:00
Filename: Rossol_Nathaniel_S_201506_PhD.pdf
Original checksum: f46c47b2f6f8e7156342c953ca07c0f7
Activity of users you follow
User Activity Date