Introduction to Machine Learning Lecture 8 Deep Belief Nets[介绍了机器学习讲座8深层信仰篮网](-52).ppt

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * The variables in h0 are conditionally independent given v0. Inference is trivial. We just multiply v0 by W transpose. The model above h0 implements a complementary prior. Multiplying v0 by W transpose gives the product of the likelihood term and the prior term. Inference in the directed net is exactly equivalent to letting a Restricted Boltzmann Machine settle to equilibrium starting at the data. Inference in a directed net with replicated weights v1 h1 v0 h0 v2 h2 etc. + + + + The learning rule for a sigmoid belief net is: With replicated weights this becomes: v1 h1 v0 h0 v2 h2 etc. First learn with all the weights tied This is exactly equivalent to learning an RBM Contrastive divergence learning is equivalent to ignoring the small derivatives contributed by the tied weights between deeper layers. Learning a deep directed network v1 h1 v0 h0 v2 h2 etc. v0 h0 Then freeze the first layer of weights in both directions and learn the remaining weights (still tied together). This is equivalent to learning another RBM, using the aggregated posterior distribution of h0 as the data. v1 h1 v0 h0 v2 h2 etc. v1 h0 What happens when the weights in higher layers become different from the weights in the first layer? The higher layers no longer implement a complementary prior. So performing inference using the frozen weights in the first layer is no longer correct. Using this incorrect inference procedure gives a variational lower bound on the log probability of the data. We lose by the slackness of the bound. The higher layers learn a prior that is closer to the aggregated posterior distribution of the first hidden layer. This improves the network’s model of the data. Hinton, Osindero and Teh (2006) prove that this