Multimodal Hand Signal and Speech Communication Classification Framework for the Construction Industry: The Case of Communication between Crane Signalman and Crane Operator

• Author / Creator

  Mansoor, Asif

• On construction sites, communication is key to boosting teamwork and improving worker performance. It allows workers to coordinate their activities, share information, and respond to potential hazards quickly and efficiently. In current practice, workers on construction sites typically rely on verbal, hand signalling, and two-way radio communication systems. Verbal and hand signalling communication are used when the workers are in close proximity to one another, whereas two-way radio communication is used when the distance is long and verbal and hand signalling communication are not possible. However, construction sites today are increasingly busy, congested, and noisy, making traditional means of communication less effective. This, in turn, results in communication errors, which can lead to disastrous accidents on construction sites. Meanwhile, the introduction in recent years of technologies such as deep learning and sensor-based approaches has resulted in a number of applications to improve safety, productivity, and surveillance. In this regard, the present research proposes a multimodal construction site communication classification system that uses innovative technologies to improve the reliability of communication on construction sites. In this research, communication between crane operator and signalman on the construction site is used as a case. The proposed framework offers a reliable, real-time communication classification system for use on construction sites as a supplementary means of communication in crane operations. Firstly, by developing computer vision-based integrated deep learning model with the capability to detect and classify dynamic hand signals in real-time, even in the presence of complex and varying weather conditions at the construction site. Secondly, by developing sensor-based smart construction glove that uses machine learning models to classify crane signalman dynamic hand signals in real-time. Additionally, to enhance speech communication, this research proposed a one-dimensional convolutional neural network model. This model is designed to identify the crane signalman speech commands in real-time by providing crane operators with the necessary keywords to understand the signalman's instructions and support their decision-making process. Finally, the proposed framework implement the concept of redundancy by utilizing ensemble models. These models combine the decisions from computer vision-based deep learning; sensor based smart construction glove; and keyword identification model using weighted average and majority voting techniques, resulting in a single, reliable decision output. Overall, this research improve the reliability of site communication by classifying the hand signals and speech commands (both individually and in the aggregate using ensemble models), classifying to a high level of specificity the hand signals and speech commands used in the communication between crane operators and signalmen.

• Subjects / Keywords

  • Deep Learning
  • Machine Learning
  • Ensemble model
  • Communication
  • Hand signals
  • crane operations
  • Construction industry

• Graduation date

  Fall 2023

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/r3-txm0-2b05

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.