对GRE写作自动化评分器e-rater评分准确性的实证研究
Accuracy of the e-rater Automated Scoring Engine in the GRE Writing Section: An Empirical Study

本研究通过分析ETS推出的ScoreItNow!付费评分服务，来探讨ETS研发的自动化阅卷程序e-rater在GRE分析性写作评分上的准确性。研究试图回答以下四个问题：(1) ScoreItNow!打分与实际考试的得分有多大差别；(2) ScoreItNow!是否能够对文章实际内容进行准确判断；(3) 文章字数对ScoreItNow!评分有多大影响；(4) ScoreItNow!作者分析工具给出的反馈是否准确。研究发现ScoreItNow!给出给出的分数和实际考试分数接近。但是，ScoreItNow!无法对文章实际内容进行准确判断。对于字数越多的文章，ScoreItNow!也越倾向于给出更高的分数。ScoreItNow!提供的作者分析工具中给出的反例准确性不令人满意。对于GRE写作教师而言，在教学过程中可以适当强调语言表达的重要性，但同时也要教授实用的写作方法，提高学生的批判性思维的能力。
This study aims to evaluate the accuracy of the e-rater automated scoring engine by investigating the scores and feedback provided by ScoreItNow! paid scoring service in the GRE analytical writing section. The study attempts to answer the following four question: (1) Is there any difference between the ScoreItNow! scores and the scores test-takers gotten from the GRE analytical writing section? (2) Is ScoreItNow! able to accurately evaluate essay content? (3) To what extent are the scores influenced by essay length? (4) Is the Writer’s Analysis Tools able to provide accurate feedback? The study found that scores provided by ScoreItNow! generally agree with the scores test-takers received from the GRE analytical writing section. However, ScoreItNow! cannot give accurate evaluation concerning the content of the essays. ScoreItNow! is more likely to give higher scores when essays are longer. The feedback in Writer’s Analysis Tools is far from satisfactory. GRE writing teachers are advised to not only emphasize the importance of language use in their teaching, but also impart useful writing skills and improve students’ ability to think critically.