Bias Algorithmic bias Critics have claimed that the computer-adaptive methodology may discourage some test takers since the question difficulty changes with performance. For example, if the test-taker is presented with remarkably easy questions halfway into the exam, they may infer that they are not performing well, which will influence their abilities as the exam continues, even though question difficulty is subjective. By contrast, standard testing methods may discourage students by giving them more difficult items earlier on. Critics have also stated that the computer-adaptive method of placing more weight on the first several questions is biased against test takers who typically perform poorly at the beginning of a test due to stress or confusion before becoming more comfortable as the exam continues. On the other hand, standard fixed-form tests could equally be said to be "biased" against students with less testing stamina since they would need to be approximately twice the length of an equivalent computer adaptive test to obtain a similar level of precision.
Implicit bias The GRE has also been subjected to the same racial bias criticisms that have been lodged against other admissions tests. In 1998,
The Journal of Blacks in Higher Education noted that the mean score for black test-takers in 1996 was 389 on the verbal section, 409 on the quantitative section, and 423 on the analytic, while white test-takers averaged 496, 538, and 564, respectively. The National Association of Test Directors Symposia in 2004 stated a belief that simple mean score differences may not constitute evidence of bias unless the populations are known to be equal in ability. A more effective, accepted, and empirical approach is the analysis of differential test functioning, which examines the differences in
item response theory curves for subgroups; the best approach for this is the DFIT framework.
Weak indicator of graduate school performance The GREs are criticized for not being a true measure of whether a student will be successful in graduate school.
Robert Sternberg (now of
Cornell University, but working at
Yale University at the time of the study), a long-time critic of modern intelligence testing in general, found the GRE general test was weakly predictive of success in graduate studies in psychology. The strongest relationship was found for the now-defunct analytical portion of the exam. The ETS published a report ("What is the Value of the GRE?") that points out the predictive value of the GRE on a student's index of success at the graduate level. The problem with earlier studies is the statistical phenomenon of restriction of range. A
correlation coefficient is sensitive to the range sampled for the test. Specifically, if only students accepted to graduate programs are studied (in Sternberg & Williams and other research), the relationship is occluded. Validity coefficients range from .30 to .45 between the GRE and both first year and overall graduate GPA in ETS' study. (p. 303). Graduate schools may be placing too much importance on standardized tests rather than on factors that more fully account for graduate school success, such as a thesis-requiring
Honours degree, prior research experience, GPAs, or work experience. While graduate schools do consider these areas, many times schools will not consider applicants that score below a current score of roughly 314 (1301 prior score). Kaplan and Saccuzzo also state that "the GRE predict[s] neither clinical skill nor even the ability to solve real-world problems" (p. 303). In 2007, a study by a university found a correlation of .30 to .45 between the GRE and both first year and overall graduate GPA. The correlation between GRE score and graduate school completion rates ranged from .11 (for the now defunct analytical section) to .39 (for the GRE subject test). Correlations with faculty ratings ranged from .35 to .50.
Historical susceptibility to cheating In May 1994,
Kaplan, Inc warned ETS, in hearings before a New York legislative committee, that the small question pool available to the computer-adaptive test made it vulnerable to cheating. ETS assured investigators that it was using multiple sets of questions and that the test was secure. This was later discovered to be incorrect. In December 1994, prompted by student reports of recycled questions, then Director of GRE Programs for Kaplan, Inc and current CEO of
Knewton, Jose Ferreira, led a team of 22 staff members deployed to 9 U.S. cities to take the exam. Kaplan, Inc then presented ETS with 150 questions, representing 70–80% of the GRE. According to early news releases, ETS appeared grateful to Stanley H. Kaplan, Inc. for identifying the security problem. However, on December 31, ETS sued Kaplan, Inc. for violation of a federal electronic communications privacy act, copyright laws, breach of contract, fraud, and a confidentiality agreement signed by test-takers on test day. On January 2, 1995, an agreement was reached out of court. Additionally, in 1994, the scoring algorithm for the computer-adaptive form of the GRE was discovered to be insecure. ETS acknowledged that Kaplan, Inc employees, led by Jose Ferreira, reverse-engineered key features of the GRE scoring algorithms. The researchers found that a test taker's performance on the first few questions of the exam had a disproportionate effect on the test taker's final score. To preserve the integrity of scores, ETS adopted a more sophisticated scoring algorithm. ==See also==