Sunday, February 22, 2004

Problems with the Computer-Based GRE

A helpful commenter pointed me to the above document detailing problems with wide-scale cheating on the computer-based GRE in Asia. I am less than surprised to learn of this development, not because I thought Asians particularly susceptible to cheating, but out of my long-held scepticism about the merits of supposedly adaptive, computer-based tests.

Last fall, New Jersey-based Educational Testing Services (ETS), the company that generates the Graduate Record Exam (GRE) as well as other secondary and post-secondary entrance exams, made two startling announcements -- that it planned to suspend its computer administered GRE General Test in China and other parts of Asia and that it was canceling the administration of its GRE Computer Science Subject test in China and India. [1] Both announcements followed reports from graduate admission departments and the media that incidents of cheating on these tests were widespread throughout China, Hong Kong, Taiwan, South Korea, and India. The GRE involves more than 400,000 students each year, 41,000 of whom are from China, making any substantiated accusation of cheating -- and the actions taken to combat it -- issues of concern to the world of academic integrity.

[............]

The first signs of cheating were noticed by graduate admission offices in the United States, who reported a sharp increase in test scores from Chinese and Korean applicants, including near perfect scores on the verbal section of the GRE. Eventually, ETS investigated the matter and found web sites that were publishing test questions from live versions of their computer-based test. The GRE Board acted immediately, requesting that ETS cease administration of the computer based GRE in the countries involved and recommending that U.S. graduate school deans be wary of high test scores from the region. It is easy to imagine how these actions by the ETS, the GRE Board and the U.S. academic community could affect those test takers from China and other countries who arrived at their scores honestly. It is equally easy to imagine the impact on all students, regardless of nationality, competing for a limited number of slots in top business schools.

One can also see how the scandal in Asia could spread quickly to the U.S. In the past, cheating on the GRE has been a problem in the U.S. [4] , and the involvement of the Internet makes containment an issue. Several sources predict seeing more evidence of cheating as U.S. students discover these Asian-language web sites.

It seems that one consequence of this has been the return of the paper-based version of the test to East Asia, a policy change I would have welcomed whatever the rationale behind it. There is just something profoundly disturbing to my sensibilities about a test in which one has no chance to see all of the questions or go back and check over one's previous responses, and how exactly can a computer possibly calibrate questions by "difficulty"? It seems to me that the ETS imagines that all questions in a set of problems must have a total ordering under a single, well-defined and totally objective axis of difficulty, an assumption I'm pretty certain is false. If only American students could enjoy the same good fortune as their Asian counterparts, and be given once more the option of taking a paper test; thank goodness I took the damn thing before the option was phased out!