They're doing it again

Just days after nobody but the intellectually blind could fail to see that NYC's school report cards are statistical junk, comes news of the DOE releasing teacher data reports.  Naturally, they don't publish details of its procedures which, by itself, commends the reports to the trash. 

But even without the documentation, from the sample supplied courtesy of the NY Times, it is obvious that these bozos do not know what they're doing.    I'm going to skate past lots of quibbles and get to the weirdest thing, which is those ranges around the teacher valued-added percentiles.  Think back when you got your SAT scores.  Did they give you a possible range?  Did your report say, you scored somewhere between the 50th and 90th percentiles, but we think you're probably close to the 70th (which, for what it's worth, is the wrong interpretation of a confidence interval)?  No, of course not, because a percentile, by definition, is a real position on a scale, not a statistical estimate.

"Value-added gain" is a statistical estimate (sort of) being composed of the difference between "actual gain" and "predicted gain,"  where the latter is a predicted value with an associated error range (or, to be precise, the sum of lots of predicted values and their associated error ranges -- it's not known whether DOE did that summation right).  So the value-added gain should have a confidence interval around it (well, not really, because none of these data come from a random sample, but let's suspend disbelief for a bit). 

My guess is DOE tried to translate an imprecise value-added measure into an imprecise percentile rank.  How they did that is truly a mystery, because there ain't no way the percentile ranges should be symmetrically centered around a midpoint, but they are.  Why is this? 

Warning: geekspeak ahead.

The value-added confidence interval should come with a point estimate plus or minus some error.  For example, 53 plus or minus 5.  Now, if 53 is above the 50th percentile, this minus 5 is going to pull you down more percentile points than the plus 5 is going to pull you up, so on the percentile scale you should get an assymetrical range around the point estimate.  Think back to ye olde bell curve.  Lots of folks clustered around the middle, so a small score shift moves you up past quite a few others.  But there aren't too many people out in the tails, so you the same small shift doesn't go as far.  Nope, symmetrical percentile ranges just don't make sense.

Darn and golly gee, my curiousity is whetted.  I really wonder how they did it.  Hope they publish details soon.