Design and Evaluation of Stochastic Computing Neural Networks for Machine Learning Applications

• Author / Creator

  Liu, Yidong

• Neural networks (NNs) are effective machine learning models that require significant hardware and energy consumption in the computing process. To implement NNs, stochastic computing (SC) has been considered to seek a trade-off between hardware efficiency and computation performance. However, most of the existing implementations are based on pre-training such that the weights are predetermined for the neurons at different layers; therefore, these implementations lack the ability to update the values of network parameters.
  In this research, we focus on the design and implementation of energy-efficient SC NNs for both training and inference. Three types of neural networks are proposed, including a multi-layer perceptron (MLP), a deep belief network (DBN), and a recurrent neural network (RNN).
  Several circuit designs are proposed to improve the hardware efficiency and performance of these networks. Using extended stochastic logic (ESL), the computation range of SC is extended from fractional numbers to real values determined by a binary representation. Reconfigurable computation units are designed to implement different types of activation functions such as the tanh function and the rectifier function. A triple modular redundancy (TMR) technique is employed for reducing the random fluctuations in SC. An SC divider based on TMR and a binary search algorithm is proposed with progressive precision for reducing the required sequence length. SC training components, such as the gradient computation and layer weight updating circuits, are proposed to implement learning algorithms, including the backward propagation in MLPs and the fast greedy learning algorithm in DBNs. An adaptive moment estimation (ADAM) circuit is designed to further improve the speed of the training process.
  
  The proposed SC networks are capable of solving the problems of classification of nonlinearly separable patterns. Incurring only a slight decrease in accuracy, the proposed designs require significantly smaller area and lower energy consumption compared to floating- and fixed-point implementations at a similar processing speed. The proposed SC circuits can be adapted into different types of NNs, so they are versatile for use in both the training and inference processes. These results show the potential of SC design techniques in machine learning applications.

• Subjects / Keywords

  • recurrent neural network
  • neural network
  • deep belief network
  • multi-layer perceptron
  • stochastic computing

• Graduation date

  Fall 2019

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/r3-sg1h-mf48

• License

  Permission is hereby granted to the University of Alberta Libraries to reproduce single copies of this thesis and to lend or sell such copies for private, scholarly or scientific research purposes only. Where the thesis is converted to, or otherwise made available in digital form, the University of Alberta will advise potential users of the thesis of these terms. The author reserves all other publication and other rights in association with the copyright in the thesis and, except as herein before provided, neither the thesis nor any substantial portion thereof may be printed or otherwise reproduced in any material form whatsoever without the author's prior written permission.