RetrievalEvaluation医疗专题知识.ppt

RetrievalEvaluation医疗专题知识

Outline

Introduction

RetrievalPerformanceEvaluation

Recallandprecision

Alternativemeasures

ReferenceCollections

TRECCollection

CACMISICollection

CFCollection

TrendsandResearchIssues

Introduction

Typeofevaluation

Functionalanalysisphase,andErroranalysisphase

Performanceevaluation

Performanceevaluation

Responsetime/spacerequired

Retrievalperformanceevaluation

Theevaluationofhowpreciseistheanswerset

RetrievalPerformanceEvaluation

評估以batchquery為主旳IR系統

collection

RelevantDocs

InAnswerSet

|Ra|

RelevantDocs

|R|

AnswerSet

|A|

Recall=|Ra|/|R|

Precision=|Ra|/|A|

Sortedbyrelevance

Precisionversusrecallcurve

Rq={d3,d5,d9,d25,d39,d44,d56,d71,d89,d123}

P=100%atR=10%

P=66%atR=20%

P=50%atR=30%

Rankingforqueryq:

Usuallybasedon11standardrecalllevels:0%,10%,...,100%

Precisionversusrecallcurve

Forasinglequery

Fig3.2

AverageOverMultipleQueries

P(r)=averageprecisionattherecalllevelr

Nq=Numberofqueriesused

Pi(r)=Theprecisionatrecalllevelrforthei-thquerInterpolatedprecision

Rq={d3,d56,d129}

P=33%atR=33%

P=25%atR=66%

P=20%atR=100%

P(rj)=maxri≦r≦rj+1P(r)

Interpolatedprecision

Letrj,j{0,1,2,…,10},beareferencetothej-thstandardrecalllevel

P(rj)=maxri≦r≦rj+1P(r)

R=30%,P3(r)~P4(r)=33%

R=40%,P4(r)~P5(r)

R=50%,P5(r)~P6(r)

R=60%,P6(r)~P7(r)=25%

Averagerecallvs.precisionfigure

SingleValueSummaries

Averageprecisionversusrecall:

Compareretrievalalgorithmsoverasetofexamplequeries

Sometimesweneedtocompareindividualquery’sperformance

Averageprecision可能會隱藏演算法中不正常旳部分

可能需要懂得,兩個演算法中，對某特定query旳performance為何

Needasinglevaluesummary

Thesinglevalueshouldbeinterpretedasasummaryofthecorrespondingprecisionversusrecallcurve

SingleValueSummaries

AveragePrecisionatSeenRelevantDocuments

Averagingtheprecisionfiguresobtainedaftereachnewrelevantdocumentisobserved.

Example:Figure3.2,(1+0.66+0