Greedy Pruning for Continually Adapting Networks

• Author / Creator

  Shah, Haseeb

• Gradient Descent algorithms suffer many problems when learning representations using fixed neural network architectures, such as reduced plasticity on non-stationary continual tasks and difficulty training sparse architectures from scratch. A common workaround is continuously adapting the neural network by generating and pruning the features, a process often called Generate and Test. This thesis focuses on neural network pruning in the online, continual setting. We look at existing pruning metrics and propose a novel pruner that attempts to estimate the ideal greedy pruner. Additionally, we observe that greedy pruning can be ineffective when features are highly correlated and does not remove these redundant features. To mitigate this issue, we also propose online feature decorrelation. Through empirical experiments in the online supervised learning setting, we show that a greedy pruner combined with the proposed feature decorrelator allows us to continually replace useless parts of the network with new features while producing a statistically significant performance improvement.

• Subjects / Keywords

  • Neural Network Pruning
  • Continual Learning
  • Representation Learning
  • Generate and Test
  • Machine Learning

• Graduation date

  Spring 2023

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-zy2w-7f31

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.