This paper describes our implementation of several neural networks built on a field programmable gate array (FPGA) and used to recognize a handwritten digit dataset—the Modified National Institute of Standards and Technology (MNIST) database. We also propose a novel hardware-friendly activation function called the dynamic Rectifid Linear Unit (ReLU)—D-ReLU function that achieves higher performance than traditional activation functions at no cost to accuracy. We built a 2-layer online training multilayer perceptron (MLP) neural network on an FPGA with varying data width. Reducing the data width from 8 to 4 bits only reduces prediction accuracy by 11%, but the FPGA area decreases by 41%. Compared to networks that use the sigmoid functions, our proposed D-ReLU function uses 24% - 41% less area with no loss to prediction accuracy. Further reducing the data width of the 3-layer networks from 8 to 4 bits, the prediction accuracies only decrease by 3% - 5%, with area being reduced by 9% - 28%. Moreover, FPGA solutions have 29 times faster execution time, even despite running at a 60× lower clock rate. Thus, FPGA implementations of neural networks offer a high-performance, low power alternative to traditional software methods, and our novel D-ReLU activation function offers additional improvements to performance and power saving.
References
[1]
Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., et al. (2017) Mastering the Game of Go without Human Knowledge. Nature, 550, 354-359. https://doi.org/10.1038/nature24270
[2]
Jouppi, N.P., Borchers, A., Boyle, R., Cantin, P., Chao, C., Clark, C., Coriell, J., et al. (2017) In-Datacenter Performance Analysis of a Tensor Processing Unit. Proceedings of the 44th Annual International Symposium on Computer Architecture, Toronto, June, 2017, 1-12. https://doi.org/10.1145/3079856.3080246
[3]
Wikipedia (2020) Rectifier (Neural Networks). https://en.wikipedia.org/wiki/Rectifier_(neural_networks)
[4]
Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) ImageNet Classification with Deep Convolutional Neural Networks. Communications of the ACM, 60, 84-90.
[5]
Maas, A.L., Hannun, A.Y. and Ng, A.Y. (2013) Rectifier Nonlinearities Improve Neural Network Acoustic Models. Proceedings of the 30th International Conference on Machine Learning, Atlanta, 16-21 June 2013, 6.
[6]
He, K.M., Zhang, X.Y., Ren, S.Q. and Sun, J. (2015) Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. 2015 IEEE International Conference on Computer Vision, Santiago, 7-13 December 2015, 1026-1034. https://doi.org/10.1109/ICCV.2015.123
[7]
Zhang, C., Li, P., Sun, G.Y., Guan, Y.J., Xiao, B.J. and Cong, J. (2015) Optimizing FPGA-Based Accelerator Design for Deep Convolutional Neural Networks. Proceedings of the 2015 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, February 2015, 161-170. https://doi.org/10.1145/2684746.2689060
[8]
Qiu, J.T., Wang, J., Yao, S., Guo, K.Y., Li, B.X., Zhou, E.J., Yu, J.C., Tang, T.Q., et al. (2016) Going Deeper with Embedded FPGA Platform for Convolutional Neural Network. 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, Monterey, February 2016, 26-35. https://doi.org/10.1145/2847263.2847265
[9]
Li, H.M., Fan, X.T., Jiao, L., Cao, W., Zhou, X.G. and Wang, L.L. (2016) A High Performance FPGA-Based Accelerator for Large-Scale Convolutional Neural Networks. 2016 26th International Conference on Field Programmable Logic and Applications, Lausanne, 29 August-2 September 2016, 1-9. https://doi.org/10.1109/FPL.2016.7577308
[10]
Lu, L.Q., Liang, Y., Xiao, Q.C. and Yan, S.G. (2017) Evaluating Fast Algorithms for Convolutional Neural Networks on FPGAs. IEEE 25th Annual International Symposium on Field-Programmable Custom Computing Machines, Napa, 30 April-2 May 2017, 101-108. https://doi.org/10.1109/FCCM.2017.64
[11]
Han, S., Mao, H. and Dally, W.J. (2015) Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. International Conference on Learning Representations 2016, San Juan, 2-4 May 2016, 1-14.
[12]
Bouvett, E., Casha, O., Grech, I., Cutajar, M., Gatt, E. and Micallef, J. (2012) An FPGA Embedded System Architecture for Handwritten Symbol Recognition. 2012 16th IEEE Mediterranean Electrotechnical Conference, Yasmine, 25-28 March 2012, 653-656. https://doi.org/10.1109/MELCON.2012.6196516
[13]
Suyyagh, A. and Abandah, G. (2013) FPGA Parallel Recognition Engine for Handwritten Arabic Words. Journal of Signal Processing Systems, 78, 163-170. https://doi.org/10.1007/s11265-013-0848-x
[14]
Huynh, T. (2014) Design Space Exploration for a Single-FPGA Handwritten Digit Recognition System. 2014 IEEE 5th International Conference on Communications and Electronics, Danang, 30 July-1 August 2014, 291-296. https://doi.org/10.1109/CCE.2014.6916717
[15]
Moradi, M., Pourmina, M.A. and Razzazi, F. (2010) FPGA-Based Farsi Handwritten Digit Recognition System. International Journal of Simulation: Systems, Science and Technology, 11, 17-22.
[16]
Nahmias, M.A., Shastri, B.J., Tait, A.N. and Prucnal, P.R. (2013) A Leaky Integrate-and-Fire Laser Neuron for Ultrafast Cognitive Computing. IEEE Journal on Selected Topics in Quantum Electronics, 19, 1-12. https://doi.org/10.1109/JSTQE.2013.2257700
[17]
Kudrolli, K., Shah, S. and Park, D. (2015) Handwritten Digit Classification on FPGA.
[18]
Amin, H., Curtis, K.M. and Hayes-Gill, B.R. (1997) Piecewise Linear Approximation Applied to Nonlinear Function of a Neural Network. IEE Proceedings—Circuits, Devices and Systems, 144, 313-317. http://dx.doi.org/10.1049/ip-cds:19971587
[19]
Wikipedia (2020) Chain Rule. https://en.wikipedia.org/wiki/Chain_rule
