AutomaticallyBuildingSpecialPurposeSearchEngineswith.ppt

Andrew McCallum, Just Research Information Extractionfrom the World Wide Web Andrew McCallum University of Massachusetts Amherst William Cohen Carnegie Mellon University Example: The Problem Example: A Solution Extracting Job Openings from the Web What is “Information Extraction” What is “Information Extraction” What is “Information Extraction” What is “Information Extraction” What is “Information Extraction” What is “Information Extraction” IE in Context Why IE from the Web? Science Grand old dream of AI: Build large KB* and reason with it. IE from the Web enables the creation of this KB. IE from the Web is a complex problem that inspires new advances in machine learning. Profit Many companies interested in leveraging data currently “locked in unstructured text on the Web”. Not yet a monopolistic winner in this space. Fun! Build tools that we researchers like to use ourselves:Cora CiteSeer, MRQE.com, FAQFinder,… See our work get used by the general public. Tutorial Outline IE History Landscape of problems and solutions Parade of models for segmenting/classifying: Sliding window Boundary finding Finite state machines Trees Overview of related problems and solutions Where to go from here IE History Pre-Web Mostly news articles De Jong’s FRUMP [1982] Hand-built system to fill Schank-style “scripts” from news wire Message Understanding Conference (MUC) DARPA [’87-’95], TIPSTER [’92-’96] Most early work dominated by hand-built models E.g. SRI’s FASTUS, hand-built FSMs. But by 1990’s, some machine learning: Lehnert, Cardie, Grishman and then HMMs: Elkan [Leek ’97], BBN [Bikel et al ’98] Web AAAI ’94 Spring Symposium on “Software Agents” Much discussion of ML applied to Web. Maes, Mitchell, Etzioni. Tom Mitchell’s WebKB, ‘96 Build KB’s from the Web. Wrapper Induction Initially hand-build, then ML: [Soderland ’96], [Kushmeric ’97],… What makes IE from the Web Different? Landscape of IE Tasks (1/4):Pattern Feature Domain Landscape of IE Tasks (2/4):Pattern Sco