In School Testing, as in Polling, "Margin of Error" Matters; Report Shows Pitfalls of Reporting Test Results for Small Groups

1/14/2004

From: Kathy Westra of the Rural School and Community Trust, 202-955-7177, Theodore Coladarci of the University of Maine, 207-942-6694

WASHINGTON, Jan. 14 -- One of the many requirements of the federal "No Child Left Behind" Act (NCLB) is the public reporting of test results to document whether students are making "adequate yearly progress" (AYP) in school. As a result, newspapers across the country are now telling parents, students, teachers, and community members how schools are doing. In many cases, schools are being labeled as "failing" under NCLB. But what does it all mean? Researcher Theodore Coladarci of the University of Maine warns that, when it comes to reporting test scores-particularly those for small schools-a little caution is in order.

"Indicators of school-level achievement," Coladarci writes, "are subject to random year-to-year variation in much the same way that the results of an opinion poll will vary from one random sample to another. This variation, which is more pronounced for a small school, should be taken into account by education officials when evaluating school progress in a policy climate of high stakes." Coladarci, a respected rural education researcher who also is editor of Journal of Research in Rural Education, discusses this problem in a new paper titled Gallup Goes to School: The Importance of Confidence Intervals for Evaluating "Adequate Yearly Progress" in Small Schools. The paper is available at no charge on the website of the Rural School and Community Trust (Rural Trust) at http://www.ruraledu.org/docs/nclb/coladarci.htm.

According to Coladarci, "the demands of NCLB often are at odds with the reality of rural education." Applying a margin of error, or confidence interval, to the reporting of test score results for small schools "is a matter of fairness," Coladarci asserts. "When a small school drops below the AYP target one year, it is quite likely that this school had a 'bad bounce' rather than a real decline due to weak instruction, poorly aligned curriculum, ineffective leadership, and the like. The question, then, is this: When a school falls short of the AYP target, with what warrant can we conclude that the school-particularly a small school-truly is not making adequate progress?" State policymakers are right to question whether a small school that fails to make the required progress under NCLB is really failing to perform, notes Marty Strange, policy program director for the Rural Trust. "Some 27 states have reached the logical-and statistically responsible-conclusion that the best way to address this problem is to attach a 'margin of error' to the percentage of students who score proficient. Gallup Goes to School explains why this issue is so important to state policymakers and makes a strong case for policies that use 'confidence intervals' in reporting test scores."

The Rural School and Community Trust (Rural Trust) is the premier national nonprofit organization addressing the crucial relationship between good schools and thriving rural communities. Working in some of the poorest, most challenging rural places, the Rural Trust involves young people in learning linked to their communities, improves the quality of teaching and school leadership, advocates for appropriate state educational policies, and addresses the critical issue of funding for rural schools.