Vision-assisted behavior-based construction safety: Integrating computer vision and natural language processing

• Author / Creator

  Wang, Yiheng

• Background: Construction sites can be hazardous places. Behavior-based safety is a method to optimize workers’ behaviors and improve site safety. Previous behavior-based safety has been criticized for their low efficiency because of manual observation. The community has conducted enormous studies about applying advanced computer vision-based methods to automate the monitoring and observation of construction sites. However, the lack of methods for extracting semantic information and identifying safety hazards from construction imagery still poses a significant challenge for the development of sophisticated vision-assisted behavior-based safety programs.
  Objectives: This research aims to automate the processes where the manual observation and inspection is needed in the traditional construction safety management by (1) enrich the information could be extracted from construction images, supporting safety hazard identification, (2) automate the safety hazard identification on site, and enable reasoning about the hazard identification according to safety regulations, and (3) automate the image records management and retrieval for efficient safety analysis.
  Methods: Firstly, this research proposes a method to extract objects, activities, and interaction information from construction images. This method utilizes image captioning techniques to generate image captions for construction images containing semantic information. Secondly, this research proposes a novel visual–text semantic similarity method to compare construction image captions with safety regulation rules, enabling automatic safety hazard identification and reasoning. Finally, this research proposed a novel content-based image retrieval method for construction image repositories-based object detection. This will help safety managers query and retrieve similar cases from monitoring image records, and conduct behavior analysis.
  Outcomes: This research will improve the current vision-based construction management applications in the following ways: (1) it helps automate the monitoring and observation of construction sites; (2) it provides an automated method to identify potential safety hazards on construction sites and give reasoning of their violation about safety rules; and (3) it provides an information retrieval system for construction image repositories, enabling fast image retrieval and case-based reasoning and analysis.

• Subjects / Keywords

  • Construction Safety
  • Deep Learning
  • Computer Vision
  • Natural Language Processing
  • Behavior-based Safety

• Graduation date

  Fall 2023

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/r3-95vf-g173

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.