The NYT reports that the Madison Wisconsin school district turned down $2 million dollars in Reading First grant money. The reason: they didn't want to use a phonics-based reading program as required by Reading First. Instead, Madison wanted to continue using the district-created "balanced literacy" reading program. Nothing wrong with that per se; Reading First is a voluntary program entailing federal oversight.
But with NCLB requirements for increased student proficiency looming on the horizon, the pressure is on for low-performing school districts, like Madison, to increase student performance. It would be quite an embarrassment if Madison's reading program failed to produce results, considering their very visible refusal to adopt a phonics-based reading program--especially since there was grant money attached.
So what do you suppose Madison did when the agenda-driven NYT came looking for a poster child of a school district bullied by the Feds in the (phony) scandal-plagued Reading First program? They cooked the books.
Under their system, the share of third graders reading at the top two levels, proficient and advanced, had risen to 82 percent by 2004, from 59 percent six years earlier, even as an influx of students in poverty, to 42 percent from 31 percent of Madison’s enrollment, could have driven down test scores. The share of Madison’s black students reading at the top levels had doubled to 64 percent in 2004 from 31 percent six years earlier.
And while 17 percent of African-Americans lacked basic reading skills when Madison started its reading effort in 1998, that number had plunged to 5 percent by 2004. The exams changed after 2004, making it impossible to compare recent results with those of 1998.
Madison's scores did rise from 58.9% in 1998 to 82.7% in 2004, an apparent rise of +0.72 standard deviations (s.d.) which is a large effect size. But the question remains: did student performance actually improve?
Let's find out.
Madison's scores rose less than Wisconsin's
According to the Wisconsin Reading Comprehension Test (3rd grade), the number of proficient students in Wisconsin rose 22.5 points from 64.9% in 1998 to 87.4% is 2005, an increase of +0.77 s.d. So it wasn't just Madison's scores that rose, scores rose across the board in Wisconsin. If anything, Madison's scores rose slightly less than the average score in Wisconsin.
But, is such an across-the-board gain in achievement realistic in the first place?
Wisconsin's NAEP scores remained flat
NAEP scores for Wisconsin show that the number of proficient (and above) students was 34% percent in 1998 and dropped slightly to 33% in 2003 and stayed there in 2005, the last time fourth graders were tested for reading. Students scoring at the basic level (and above) dropped from 69% to 67% during this same period. From the period 1992-2005, the achievement gap between black and white students rose from 28 points to 33 points and the gap between poor and non-poor students dropped slightly from 28 points to 25 points.
So, NAEP shows us that the reading performance of Wisconsin fourth graders has basically remained flat since about 2000. (Go here and select Wisconsin as the jurisdiction.)
The NAEP scores tell us that Wisconsin's miraculous gains in reading achievement from 1998-2005 are non-existent. Wisconsin did what most states did in response to NCLB, they goosed the tests to artificially increase test scores to give the appearance of increased student achievement.
Let's recalibrate Wisconsin's gain of +0.77 s.d. to 0.0 s.d. to account for the non-gains made in NAEP. This means that Madison's real performance during 1998-2005 actually declined by -0.05 s.d.
Madison's schools eligible for Reading First funding performed significantly below Madison's other schools
According to this source, there were four Madison schools eligible for Reading First funding: Glendale, Hawthorne, Lincoln, and Orchard Ridge. These schools never received this funding because Madison choose to stick with its Balanced Literacy reading program. Let's see how those schools performed.
The average gain made by these four schools was only 21.6% or +0.56 s.d. This is a significant under-performance compared to the statewide gain of +0.77 s.d. Using our NAEP recalibration, the performance of these schools actually declined by -0.19 s.d. That's big.
In contrast, the average gain made by the remaining schools in Madison was 25.4% or +0.80 s.d. which is about the same gain made statewide.
The Disaggregated Data
In the NYT article, Madison officials made some wild suggestions about the relative performance of black students compared to other students (mostly white) in Madison. I showed in my previous post how these percentile gains are misleading due to how scores are distributed (a normal distribution), so I won't repeat it here.
In fact, I believe the scores that Madison is using are inaccurate. Madison's numbers have black performance rising by +0.86 s.d., which is high. I can't find disaggregated data for the WRCT, but the disaggregated data from the 3rd grade performance on the new WSAS test shows that black performance is significantly less than this and less than the average gain made by white students. It's not an apples to apples comparison, but it is consistent with the rest of the data.
Madison is cooking the books.
Its schools slightly underperformed Wisconsin schools and Madison's other schools.
In fact, NAEP data shows that the gains made by Wisconsin are illusory. It's doubtful that scores rose at all in Wisconsin.
If we look at only the schools in Madison that were eligible for Reading First funding, we see that these schools performed significantly worse than other schools in Wisconsin.
So it appears that Madison's Balanced Literacy reading program, which cost the district $2 million, failed to increase student performance in Madison and actually caused a relative decline in the schools that were supposed to get Reading First funding.
This is exactly what we expect to see in your typical balanced literacy program, at-risk children failing to achieve. These are the children most damaged by "balanced literacy" programs, kids with low language skills and background knowledge. These were the kids that Reading First intended to serve.