Obsolete Methods Cause Researchers To Miss Significant Discoveries, USC Psychologist Says

In a provocative new paper, a University of Southern California quantitative psychologist says many psychology researchers are using obsolete statistical techniques and as a result are missing important discoveries.

"Psychology journals are littered with nonsignificant results that would have been significant if a more modern method had been used," says Rand R. Wilcox, Ph.D., writing in the March issue of American Psychologist.

Dr. Wilcox, a professor of psychology, a statistician and the author of several books on statistics and social science research, suggests that psychology researchers should revisit many older studies and re-analyze the data using modern computational techniques. Furthermore, he says, psychology education must be upgraded so that psychologists are aware of the new techniques.

"It's in everybody's best interest to take advantage of modern technology," says Wilcox, who teaches statistical analysis to psychology majors in the USC College of Letters, Arts and Sciences. "Computers today can easily do calculations that were impossible 50 or even five years ago."

As cutting-edge statisticians have developed increasingly sophisticated techniques for handling data, Wilcox says, "there's an ever-widening gap between modern statistical methods and the techniques psychologists are continuing to use."

Typically, psychology studies involve comparisons of two or more groups of subjects, with researchers collecting data on individuals and looking at the variables, Wilcox explains. The classic analytical tool familiar to every psychologist is the T test, in which the average, or mean, of each group is compared to the others'.

The T test is easy to understand and easy to calculate, and, says Wilcox, it can easily lead to the wrong conclusion. For example, if two groups of people each have an average IQ of 100, you might conclude from the T test that there is no significant intelligence difference between the two groups.

But what if every individual in the first group had an IQ very close to 100 while the second group included some very intelligent people with IQs of 125 or more and some very unintelligent people with IQs of 75 or less? Then the two groups can have differences that the T test would not reveal.

Although this may seem a gross simplification, many researchers have essentially made this mistake with the T test, Wilcox says.

When researchers collect data for a study, they usually find some values that are unusually large or small. In the example above, one IQ might have been as high as 180 and one as low as 50. Such data are called outliers, and while they may be interesting by themselves, they wreak havoc with standard statistical methods.

Leave them in, and the outliers can skew a sample. Leave them out and apply standard methods to the data that remain -- a common strategy -- and serious technical problems may result. Outliers can be handled most effectively, Wilcox says, by carefully trimming the outliers and by applying theoretically correct statistical methods.

Wilcox says researchers have long ignored slight differences from a normal curve because they cannot handle them mathematically. Again, consider IQ as an example. Researchers assume that if you graphically plotted the IQ scores of a random sample of people, their scores would probably fall into an almost perfect bell-shaped curve. Most people are of average intelligence and their scores would be plotted at the top of the bell. The less intelligence individuals had, the farther down on the left-hand descending side of the bell their scores would fall; while the more intelligence they had, the farther down their scores would be on the right-hand descending side.

For a long time, psychologists have known that the IQ scores of a small number of extremely intelligent people can skew the upper-right side of the bell curve by pulling it out to form a little tail.

Many psychology researchers have pretended this slight distortion just doesn't occur, but it can make a real difference when looking for relationships between two groups, Wilcox points out.

In a famous study done 30 years ago, a psychology researcher compared the intelligence of two groups of children. The teachers of the children in the first group were told they could expect their students to do well on IQ tests. The teachers of the second group were told nothing about their students' test- score potential.

For decades, psychology researchers have questioned whether the children for whom positive expectancies had been suggested actually scored higher on subsequent IQ tests. Differences between the two groups were not easily studied or understood by using conventional methods. "Using modern statistical analysis and taking pretest scores into account, there is no compelling reason to believe that expectancies influence IQ test scores," Wilcox says.

Wilcox believes the problem of using old computational technique is not limited to the field of psychology but probably pervades many other research fields as well. Nor is the problem confined to research alone.

"Some of the most popular statistical software packages available today are four generations out of date," he says.

BC.WILCOX -USC- MARCH 9, 1998