Intrusion Detection Based on Reinforcement Learning

• Author / Creator

  Yang, Bin

• Powered by advancements of information and Internet technologies, there has been a rapid development in network based applications in recent years. Meanwhile, it is recognized that more attentions need to be paid to the issue of cybersecurity. The security of the network environment plays a vital role in stable functioning of the society.

  Research on cybersecurity has become more active lately. Researchers have proposed a number of approaches to protect the network. Among them, a broadly practiced approach is the intrusion detection system (IDS). Building a powerful and robust intrusion detection system is long-established as it can provide effective protection to prevent Internet from intrusions and attacks. Through pattern or rules matching, the intrusion detection system can filter out harmful traffics. However, traditional rule-based intrusion detection systems are unqualified to acclimate to the ever-changing network environments because rules are drawn up manually. Thus, a significant number of research works are focused on the development of novel methods to handle the new challenges. Benefiting from the vigorous development of machine learning and artificial intelligence, researchers have been actively deploying these new technologies to handle network traffic analytics, data processing and feature engineering, which are important modules in building the intrusion detection system. Machine learning techniques have already achieved substantial success in the area of cybersecurity.

  Reinforcement learning (RL) is one of the most significant and compelling methodologies of machine learning. It is used to describe and solve the problem of the agent in the process of interaction with the environment through learning the strategies to maximize returns or achieve specific goals. RL has achieved considerable accomplishments in a multitude of fields, such as games, robotics and autonomous systems. For example, RL has been shown as the most promising method in designing game AI agents. In some games, RL agents have outcompeted top professional players. Inspired by the success of RL in other areas, in this work, we intend to study how it can be utilized in designing the intrusion detection system to improve the cybersecurity.

  This thesis is mainly divided into two parts. Chapter 3 includes an empirical study of Proximal Policy Optimization Algorithm (PPO), one of the most well-known reinforcement learning algorithms. During this empirical study, we can further understand how RL algorithms work in game AI, and hope to find common ideas between game AI and the cybersecurity research. This way, we can apply the reinforcement learning framework to solve the intrusion detection tasks. Hence, the second part is focused on designing intrusion detection systems (IDS) based on reinforcement learning. The approaches are categorized into packet-based and flow-based, which are separately addressed in Chapter 4 and Chapter 5. In this part, network traffic data are firstly processed using different methods, and subsequently, reinforcement learning algorithms are developed as the overall framework of the intrusion detection system.

• Subjects / Keywords

  • Reinforcement Learning
  • Intrusion Detection
  • Cybersecurity

• Graduation date

  Fall 2021

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-9n3w-yc50

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.