MachineLearningforInformationExtractionAnOverview.ppt

Andrew McCallum, Just Research 15-505: Lecture 11Generative Models for Text Classification and Information Extraction Kamal Nigam Text Classification by Example Text Classification by Example Text Classification by Example Text Classification by Example Text Classification by Example How could you build a text classifier? Take some ideas from machine learning Supervised learning setting Examples of each class (a few or thousands) Take some ideas from machine translation Generative models Language models Simplify each and stir thoroughly Basic Approach of Generative Modeling Pick representation for data Write down probabilistic generative model Estimate model parameters with training data Turn model around to calculate unknown values for new data Na?ve Bayes: Bag of Words Representation Na?ve Bayes: Mixture of Multinomials Model Pick the class: P(class) For every word, pick from the class urn: P(word|class) Na?ve Bayes: Estimating Parameters Just like estimating biased coin flip probabilities Estimate MAP word probabilities: Estimate MAP class priors: Na?ve Bayes: Performing Classification Word independence assumption Take the class with the highest probability Classification Tricks of the Trade Stemming run, runs, running, ran ? run table, tables, tabled ? table computer, compute, computing ? compute Stopwords Very frequent function words generally uninformative if, in, the, like, … Information gain feature selection Keep just most indicative words in the vocabulary Na?ve Bayes Rules of Thumb Need hundreds of labeled examples per class for good performance (~85% accuracy) Stemming and stopwords may or may not help Feature selection may or may not help Predicted probabilities will be very extreme Use sum of logs instead of multiplying probabilities for underflow prevention Coding this up is trivial, either as a mapreduce or not Information Extraction with Generative Models Example: A Problem Example: A Solution Extracting Job Openings from