From these data, we cannot say that the Reading First districts did or did not increase their proficiencies. There just aren't enough data. But — and this is crucial — neither can the Reading First districts claim that they raised their proficiencies using these data. The only way we can determine whether these school systems did or did not raise their proficiencies is by analyzing the raw data, and not the aggregates by school district. In other words, Ken was right, and the journalist was wrong.
I like that last sentence so much I'm going to repeat it.
In other words, Ken was right, and the journalist was wrong.
I did a very quick initial analysis by assuming both distributions were essentially normal so I wouldn't have to fix the improperly formatted data set. RWP, an expert in data analysis, did the hard word of reformatting the data set and did a complete analysis. I re-analyzed the data and came up with the same numbers as RWP. Time permitting, I'll update my figures with the re-analyzed numbers. The effect sizes will change slightly, but my conclusions will still hold.
The important thing to remember is that RWP's analysis includes the following assumption.
We are assuming here either that the proficiency exam standards did not change between the two years or that the proficiency reports for the two years are comparable (if they are not, then Wisconsin cannot make any statement about their proficiency levels over time).
As the NAEP data clearly shows, the Wisconsin's proficiency exam standards did change between 1998 and 2005. NAEP scores declined slightly, while the Wisconsin scores magically skyrocketed. Suspiciously so.
So, when RWP says that Wisconsin and Madison can validly claim that scores have increased, this conclusion only holds if Wisconsin's proficiency exam standards didn't change between 1998 and 2005. And, as we know from NAEP scores, they did.
RWP's analysis reveals an importnt factoid.
So far, everything looks positive, until we look at the skewness. Go back to that bell curve water balloon. If you grab the tail on the right and pull it further to the right, more of the water will spill into that tail, right? That's what we call a right-skewed curve, and it has a positive skewness factor. If you pull the left tail out, more water spills into that left tail, and we have a left-skewed curve, with a negative skewness factor. When we look at the skewness for the two years, both are left-skewed — that is, in both years, there are more data in the left tails (less than proficient) — but the 04-05 curve is more left-skewed than 98-99 (-1.86 and -0.92, respectively). So even though it does look like Wisconsin may have improved the reading proficiency between the two years, they also slightly increased those who were less than proficient (if this seems like a paradox to you, think of the water balloon again, and all will be made clear).
Even though Wisconsin's scores magically skyrocketed, the number if students who were less than proficient increased slightly. Kids at top learn more, but the kids at the bottom continue to struggle. These would be the kids that Reading First aimed to serve.
I think we can finally put this one to bed, unless Madison officials decide to make more outlandish claims.