工业工程专业英语（第11章）.ppt

(四) Evaluation of the Performance 1. Characteristics of Good Measures Measures are based on the performance criteria identified. A good measure is one that reflects the essence of the criterion and that implies a unit for quantification. Validity and reliability are the two most important qualities of measures. Validity relates to the accuracy and represent ativeness of the data measured. It is the property linking what is actually measured to what the concept purports to measure. In other words, the measure should reflect the real meaning of the concept to be measured. Reliability is the degree of stability or consistency of the data collected over time. That is, if the same data collection procedure is repeated under the same conditions, would the same scores be obtained? It is the property of stability in recording the results of measuring. It is directly concerned with the accuracy and precision of the measuring instrument. 2. Measure units The two most well known performance measures are recall and precision. Recall measures the proportion of the relevant documents actually retrieved. That is, assuming that the collection contains 100 items on the subject being sought, how many of the 100 have been retrieved? Precision relates to the percentage of relevant documents contained in the retrieved set. In other words, when one retrieves 50 citations, how many of these are actually relevant? Naturally, it is assumed that relevance has been established by someone or some yardstick. No matter how limited these measures are, recall and precision have become the standards used by most evaluation experiments In probabilistic terms, recall is an estimation of the conditional probability that an item will be retrieved given that it is relevant to the query. Precision is an estimation of the conditional probability that an item will be relevant given that it is retrieved5. To take account of the size of the collection from which retrieval is