Extending Differentiable Programming to include Non-differentiable Modules using Differentiable Bypass for Combining Convolutional Neural Networks and Dynamic Programming into an End-to-end Trainable Framework

• End-to-End Learning of Dynamic Programming and Convolutional Neural Networks using Differentiable Bypass

• Author / Creator

  Nguyen, Nhat

• Differentiable Programming is the paradigm where different functions or modules are combined into a unified pipeline with the purpose of applying end-to-end learning or optimization. A natural impediment is the non-differentiability characteristic of many modules. This thesis proposes a new way to overcome this obstacle by using a concept called Differentiable Bypass (DiffBypass). DiffBypass exploits the Universal Function Approximation property of neural networks to mimic the output of non-differentiable functions or modules in the pipeline, rerouting the gradient path to bypass these components entirely.Further, as a significant application, we demonstrate the use of DiffBypass to combine Convolutional Neural Networks (CNN) and Dynamic Programming (DP) in end-to-end learning for segmenting left ventricle from short axis view of heart MRI. Our experiments show that end-to-end combination of CNN and DP requires fewer labeled images to achieve a significantly better segmentation accuracy than using only CNN by allowing the incorporation of strong prior knowledge into the pipeline to cope with lack of training data. Comparison between DiffBypass and Evolution Strategy (ES), another method that can be used to train non-differentiable modules, shows that DiffBypass is more robust and has better performance for high-dimension problems.Finally, as a technical contribution, we provide a set of recommendations for training non-differentiable modules using DiffBypass. Furthermore, we also provide a code base for reproducibility. We think DiffBypass has the potential to become a blueprint to expand differentiable programming to include non-differentiable modules.

• Subjects / Keywords

  • Semantic Segmentation
  • Convolutional Neural Networks
  • Medical Imaging
  • Differentiable Programming
  • End-to-end Learning
  • Dynamic Programming

• Graduation date

  Spring 2019

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-xkzz-k966

• License

  Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.