딥러닝 프레임워크의 비교 = 티아노, 텐서플로, CNTK를 중심으로

다국어 초록 (Multilingual Abstract)

The deep learning framework is software designed to help develop deep learning models. Some of its important functions include “automatic differentiation” and “utilization of GPU”. The list of popular deep learning framework includes Caffe (BVLC) and Theano (University of Montreal). And recently, Microsofts deep learning framework, Microsoft Cognitive Toolkit, was released as open-source license, following Google’s Tensorflow a year earlier. The early deep learning frameworks have been developed mainly for research at universities. Beginning with the inception of Tensorflow, however, it seems that companies such as Microsoft and Facebook have started to join the competition of framework development. Given the trend, Google and other companies are expected to continue investing in the deep learning framework to bring forward the initiative in the artificial intelligence business. From this point of view, we think it is a good time to compare some of deep learning frameworks. So we compare three deep learning frameworks which can be used as a Python library. Those are Googles Tensorflow, Microsoft’s CNTK, and Theano which is sort of a predecessor of the preceding two.
The most common and important function of deep learning frameworks is the ability to perform automatic differentiation. Basically all the mathematical expressions of deep learning models can be represented as computational graphs, which consist of nodes and edges. Partial derivatives on each edge of a computational graph can then be obtained. With the partial derivatives, we can let software compute differentiation of any node with respect to any variable by utilizing chain rule of Calculus.
First of all, the convenience of coding is in the order of CNTK, Tensorflow, and Theano. The criterion is simply based on the lengths of the codes and the learning curve and the ease of coding are not the main concern. According to the criteria, Theano was the most difficult to implement with, and CNTK and Tensorflow were somewhat easier. With Tensorflow, we need to define weight variables and biases explicitly. The reason that CNTK and Tensorflow are easier to implement with is that those frameworks provide us with more abstraction than Theano. We, however, need to mention that low-level coding is not always bad. It gives us flexibility of coding. With the low-level coding such as in Theano, we can implement and test any new deep learning models or any new search methods that we can think of.
The assessment of the execution speed of each framework is that there is not meaningful difference. According to the experiment, execution speeds of Theano and Tensorflow are very similar, although the experiment was limited to a CNN model. In the case of CNTK, the experimental environment was not maintained as the same. The code written in CNTK has to be run in PC environment without GPU where codes execute as much as 50 times slower than with GPU. But we concluded that the difference of execution speed was within the range of variation caused by the different hardware setup.
In this study, we compared three types of deep learning framework: Theano, Tensorflow, and CNTK. According to Wikipedia, there are 12 available deep learning frameworks. And 15 different attributes differentiate each framework. Some of the important attributes would include interface language (Python, C ++, Java, etc.) and the availability of libraries on various deep learning models such as CNN, RNN, DBN, and etc. And if a user implements a large scale deep learning model, it will also be important to support multiple GPU or multiple servers. Also, if you are learning the deep learning model, it would also be important if there are enough examples and references.

참고문헌 (Reference)

1 "https://www.tensorflow.org/"

2 "https://www.microway.com/hpc-tech-tips/deep-learning-frameworks-survey-tensorflow-torch-theano-caffe-neon-ibm-machine-learning-stack/"

3 "https://www.cs.toronto.edu/~kriz/cifar.html"

4 "https://www.cntk.ai/pythondocs/index.html"

5 "https://keras.io/"

6 "https://github.com/jcjohnson/cnn-benchmarks"

7 "https://github.com/Microsoft/CNTK"

8 "https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software"

9 "https://deeplearning4j.org/"

10 "https://colah.github.io/posts/2015-08-Backprop/"

11 "http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-tornns/"

12 "http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html"

13 "http://neuralnetworksanddeeplearning.com/"

14 "http://deeplearning.net/software/theano/index.html"

15 Bergstra, J., "Theano: A CPU and GPU Math Expression Compiler" 2010

16 Goodfellow, I., "Deep Learning" MIT Press 2016

17 Bahrampour, S., "Comparative Study of Deep Learning Software Frameworks"

18 Yu, D., "An Introduction to Computational Networks and the Computational Network Toolkit" Microsoft Research 2014