Large Scale Deep Learning.pdf

Large Scale Deep Learning Quoc V. Le Google CMU Deep Learning ?? Google is using Machine Learning ?? Machine Learning is difficult ?? Requires domain knowledge from human experts Deep Learning: ?? Great performances for many problems ?? Works well with a large amount of data ?? Requires less domain knowledge Focus: ?? Scale deep learning to bigger models and bigger problems Quoc V. Le Deep Learning ?? Google is using Machine Learning ?? Machine Learning is difficult ?? Requires domain knowledge from human experts Deep Learning: ?? Great performances for many problems ?? Works well with a large amount of data ?? Requires less domain knowledge Focus: ?? Scale deep learning to bigger models and bigger problems Quoc V. Le Quoc V. Le What is Deep Learning? Quoc V. Le x v = g(B u) … A (images, audio, texts, etc.) u = g(A x) What is Deep Learning? B Quoc V. Le x v = g(B u) … A (images, audio, texts, etc.) u = g(A x) What is Deep Learning? B Quoc V. Le … Pixels High-level features by Deep Learning Edge detectors Face detector, Cat detector Model Training Data Quoc V. Le Google’s DistBelief Goal: Train deep learning on many machines Model: A multiple layered architecture Forward pass to compute the features Backward pass to compute the gradient Model DistBelief distributes a model across multiple machines and multiple cores. Training Data Machine (Model Partition) Quoc V. Le Model partition with DistBelief Model Machine (Model Partition) Core Training Data Quoc V. Le DistBelief distributes a model across multiple machines and cores. Model partition with DistBelief Model Training Data Stochastic Gradient Descent (SGD) Model parameters are partitioned Can use up to 1000 cores Quoc V. Le Model partition with DistBelief Model Training Data But training is still slow on large data sets Can we add more parallelism? Idea: Train multiple models on different partition