1、EM Algorithm,主講人：虞台文大同大學資工所智慧型多媒體研究室,Contents,IntroductionExample Missing DataExample Mixed AttributesExample MixtureMain BodyMixture ModelEM-Algorithm on GMM,EM Algorithm,Introduction大同大學資工所智慧型多媒體研究室,Introduction,EM is typically used to compute maximum likelihood estimates given incomplete samples.
2、The EM algorithm estimates the parameters of a model iteratively.Starting from some initial guess, each iteration consists of an E step (Expectation step) an M step (Maximization step),Applications,Filling in missing data in samplesDiscovering the value of latent variablesEstimating the parameters o
3、f HMMsEstimating parameters of finite mixturesUnsupervised learning of clusters,EM Algorithm,Example:Missing Data大同大學資工所智慧型多媒體研究室,Univariate Normal Sample,Sampling,Maximum Likelihood,Sampling,Given x, it is a function of and 2,We want to maximize it.,Log-Likelihood Function,Maximizethis instead,By s
4、etting,and,Max. the Log-Likelihood Function,Max. the Log-Likelihood Function,Miss Data,Sampling,Missing data,E-Step,Let,be the estimated parameters at the initial of the tth iterations,E-Step,Let,be the estimated parameters at the initial of the tth iterations,M-Step,Let,be the estimated parameters
5、at the initial of the tth iterations,Exercise,EM Algorithm,Example:Mixed Attributes大同大學資工所智慧型多媒體研究室,Multinomial Population,Sampling,N samples,Maximum Likelihood,Sampling,N samples,Maximum Likelihood,Sampling,N samples,We want to maximize it.,Log-Likelihood,Mixed Attributes,Sampling,N samples,x3 is n
6、ot available,E-Step,Sampling,N samples,x3 is not available,Given (t), what can you say about x3?,M-Step,Exercise,Estimate using different initial conditions?,EM Algorithm,Example: Mixture大同大學資工所智慧型多媒體研究室,Binomial/Poison Mixture,# Obasongs,n0,# Children,M : married obasong,X : # Children,Binomial/Poi
7、son Mixture,# Obasongs,n0,# Children,M : married obasong,X : # Children,nA : # married Obs nB : # unmarried Obs,Unobserved data:,Binomial/Poison Mixture,# Obasongs,n0,# Children,M : married obasong,X : # Children,Completedata,Binomial/Poison Mixture,# Obasongs,n0,# Children,Completedata,Complete Dat
8、a Likelihood,# Obasongs,n0,# Children,Completedata,Complete Data Likelihood,# Obasongs,n0,# Children,Completedata,Log-Likelihood,Maximization,Maximization,E-Step,Given,M-Step,Example,EM Algorithm,Main Body大同大學資工所智慧型多媒體研究室,Maximum Likelihood,Latent Variables,Incomplete Data,Complete Data,Complete Dat
9、a Likelihood,Complete Data,Complete Data Likelihood,Complete Data,A function of parameter ,A function of latent variable Yand parameter ,If we are given ,Computable,The result is in term of random variable Y.,A function of random variable Y.,Expectation Step,Let (i1) be the parameter vector obtained
10、 at the (i1)th step.,Define,Maximization Step,Let (i1) be the parameter vector obtained at the (i1)th step.,Define,EM Algorithm,Mixture Model大同大學資工所智慧型多媒體研究室,Mixture Models,If there is a reason to believe that a data set is comprised of several distinct populations, a mixture model can be used.It ha
11、s the following form:,with,Mixture Models,Let yi1, M represents the source that generates the data.,Mixture Models,Let yi1, M represents the source that generates the data.,Mixture Models,Mixture Models,Mixture Models,Given x and , the conditional density of y can be computed.,Complete-Data Likeliho
12、od Function,Expectation,g: Guess,Expectation,g: Guess,Expectation,Zero when yi l,Expectation,Expectation,Expectation,1,Maximization,Given the initial guess g,We want to find , to maximize the above expectation.,In fact, iteratively.,The GMM (Guassian Mixture Model),Guassian model of a d-dimensional
13、source, say j :,GMM with M sources:,EM Algorithm,EM-Algorithm on GMM大同大學資工所智慧型多媒體研究室,Goal,Mixture Model,subject to,To maximize:,Goal,Mixture Model,subject to,To maximize:,Correlated with l only.,Correlated with l only.,Finding l,Due to the constraint on ls, we introduce Lagrange Multiplier , and sol
14、ve the following equation.,Finding l,1,N,1,Finding l,Finding l,Only need to maximizethis term,Consider GMM,unrelated,Finding l,Only need to maximizethis term,Therefore, we want to maximize:,unrelated,How?,knowledge on matrix algebra is needed.,Finding l,Therefore, we want to maximize:,Summary,EM alg
15、orithm for GMM,Given an initial guess g, find new as follows,Not converge,Demonstration,EM algorithm for Mixture models,Exercises,Write a program to generate multidimensional Gaussian distribution.Draw the distribution for 2-dim data.Write a program to generate GMM.Write EM-algorithm to analyze GMM data.Study more EM-algorithm for mixture.Find applications for EM-algorithm.,
