Smith was the data reporter for the L.A. Times’ controversial “Grading the Teachers” project, which ranked thousands of teachers (by name) using student test scores as a measure of achievement. He spoke with EWA about the project and responded to a new report critical of newspapers that identify individual teachers when reporting evaluation scores.
1. What was the initial reaction from teachers to the Times’ project?
Most of the teachers who called were angry that we had published individual names, but we also got calls from teachers who thanked us for giving them some feedback that they had never received before. I also heard from teachers who were upset because their colleagues had a rating, but they didn’t get one. That was one of the reasons why in the second year (of the project), I insisted that we include all teachers.
2. In a new issue brief, Center for American Progress researcher Diana Epstein suggested the L.A. Times project was a simplistic view of teacher performance. What is your response to that?
The project was not simplistic at all – in fact, it was sophisticated. You can see that on our website. We showed each teacher’s position on a distribution of the whole teaching force. You could see where their estimates fell, how many other teachers fell in the same distribution point and how far they were from the high end and the low end of the scale. You could also see their confidence intervals, and also view another page that would show how a particular teacher would rank if different evaluation models were used. That allowed readers to see how a teacher would fare in an evaluation model with no adjustment at all–just straight gain scores. It is one-dimensional in that it’s how teachers fare at improving test scores, but that’s a dimension many parents think is the most important one.
3. Epstein’s report also suggested that by publishing the names of individual teachers, newspapers were doing more harm than good. Did you weigh those risks?
My biggest criticism of the Epstein report was that it was doctrinal. It said that young people would be discouraged from aspiring to be teachers, and there would be negative consequences to this kind of reporting. But it’s not a known fact; it’s just what she thinks would happen.
The question is what did happen. At this point, we don’t see that any negative consequences have occurred. The district’s test scores increased by about the same amount they have in previous years. There was no significant difference. There were no reports of parents storming the schools and demanding different teachers for their children, throwing the campuses into turmoil.
We certainly know a lot of teachers were upset. There could have been a morale issue; I won’t disagree with that. But if we observed a clear and distinct negative impact, we would probably rethink what we’re doing. But so far, we haven’t seen that.
4. You’ve mentioned that the decision to undertake the project came after much debate and discussion in your newsroom. What were some of the issues you were wrestling with at a personal and professional level?
I measure what I do against whether it’s valuable to my readers. In this case, that was a very difficult thing to decide. I’ve grown accustomed over the years to thinking of teachers as a core part of our readership. I knew going in we were going to alienate teachers by doing this report. That was a concept that I found very hard to resolve. We would be angering a significant part of our really loyal readers. Finally, however, I had to reach the conclusion that the teachers I was used to hearing from were the ones who would actually appreciate having this information. I have no way of knowing if that’s true, but I hope it is the case.
5. What opportunities were teachers provided to respond to the project prior to publication?
When we write something about somebody, we always give them the chance to tell what they know about it, to comment or to say that we’re wrong and to give us additional information. We knew we couldn’t do this project unless we provided the teachers that opportunity. We brought in six temporary people to handle the calls, and we created an online platform where teachers could email us and tell us they wanted to see their results page and then comment on it.
We got several hundred comments – still a very small number – but most were complaints. About a half-dozen teachers used the opportunity to write about what teaching meant to them, and some of the remarks were really beautiful. But most teachers didn’t reach out. We were prepared for a huge wave of response that just didn’t happen.
I probably talked to three or four dozen teachers, some of whom thanked us for the information, and others said they were disappointed and that they didn’t believe the data represented all they had to offer their students. I encouraged them to write a comment on their own pages, but they would tell me they had been told by the teachers union not to respond. They were afraid to comment.
There was a chance to develop a really unique literature of teachers talking about what they do in connection with these scores, and the union cut it off. That was definitely one of my biggest disappointments.
