Parameters Driving Effectiveness of Automated Essay Scoring with LSA.pdfVIP

PARAMETERS DRIVING EFFEC- TIVENESS OF AUTOMATED ESSAY SCORING WITH LSA Fridolin Wild, Christina Stahl, Gerald Stermsek and Gustaf Neumann Parameters Driving Effectiveness of Automated Essay Scoring with LSA Fridolin Wild, Christina Stahl, Gerald Stermsek, Gustaf Neumann Department of Information Systems and New Media, Vienna University of Economics and Business Administration, Augasse 2-6, A-1090 Vienna, Austria {firstname.lastname}@wu-wien.ac.at Abstract Automated essay scoring with latent semantic analysis (LSA) has recently been subject to increasing interest. Although previous authors have achieved grade ranges similar to those awarded by humans, it is still not clear which and how parameters improve or decrease the effectiveness of LSA. This pa- per presents an analysis of the effects of these parameters, such as text pre- processing, weighting, singular value dimensionality and type of similarity measure, and benchmarks this effectiveness by comparing machine-assigned with human-assigned scores in a real-world case. We show that each of the identified factors significantly influences the quality of automated essay scor- ing and that the factors are not independent of each other. Introduction Computer assisted assessment in education has a long tradition (cf. Hearst, 2000). Early experiments on grading free text responses were conducted dur- ing the Project Essay Grade (PEG) by Page (Page, 1966) in 1966. Page used punched cards and based his experiments predominantly on syntactical fea- tures. Research today focuses on emulating a human-semantic understand- ing, backed up by hitherto unknown computing power. Semantic understand- ing in automated essay scoring is required in order to automatically evaluate the content of written essays on the