After 25 Years, No Shortage of Charter Schools Research

Do charter schools really outperform traditional public schools? Is any such comparison skewed by the caliber of students who attend charters and their district-run counterparts?

Few questions are more contested in education policy circles, but 25 years since the first charter school opened in Minnesota, there is more data available than ever to find answers. Two prominent education researchers waded into the debate over charter schools and charter research at a recent Education Writers Association seminar in Los Angeles.

The panel’s moderator — Sarah Sparks, who covers education research for Education Week — started the discussion by highlighting the significance for research at the 25-year milestone.

“We’re just at the point now where we are starting to get regular data,” said Sparks, adding a note of caution for reporters given the ferocity of the charter debate. “But we have to go above and beyond what we do when covering educational research.”

The difficulty of covering charter school research is underscored by the work of the two panelists. Macke Raymond is the executive director of the Center for Research on Education Outcomes (CREDO) at Stanford University. Over the years, CREDO has found that while there are many high-flying charters, the movement as a whole is still plagued by far too many poor performers.

One of CREDO’s best-known studies, from 2015, examined charter achievement in 41 urban communities in 22 states. It found that, in the aggregate, charters outperformed similar  district-run schools in both reading and math, based on gains in student test scores. However, those national averages don’t tell the whole story. In math, for instance, charters outperformed district-run schools in 26 out of 41 cities, but trailed district schools in 11 other cities. In four urban locations math gains were about the same in both types of public schools.

A critic of many charter school studies joined Raymond on the panel. Kevin Welner, the director of the National Education Policy Center at the University of Colorado Boulder, questions the validity of any charter school study that doesn’t account for the fact that, whether intentionally or not, charter schools tend to draw students with parents who are more engaged in their children’s learning and those who have the time and resources to navigate the school choice landscape.

‘Virtual Twins’

Raymond touts CREDO’s methodological approach, which creates so-called “virtual twins.” That is, they use prior performance data to compare the achievement of students who enrolled in charter schools to similarly situated students who didn’t take advantage of charters but instead remained at the traditional public school in which the virtual twin at the charter school opted out. CREDO uses prior data to create baseline test scores for each twin and then compares where they go from there.

For his part, Welner has recently examined which students access charter schools. He concludes that too many highly regarded charters have created systems that dissuade families with the greatest needs from enrolling.

“Parents make choices, but schools make choices too,” said Welner. “Some require families to attend getting-to-know-you sessions, some require parents to write several essays. I’ve even seen a school require a three-page autobiographical essay. So charters create all of these incentives and disincentives that attract and enroll students whose behavior matches the school’s ideal.”

Welner argues that all of these policies taken together not only undermine the ability to compare traditional public and charter schools, but also undermine the very societal value of public education.

Raymond, however, said her own research does not bear out suggestions of “cream skimming” by charters, noting that a forthcoming study later this year “finds no evidence for it in “almost all of the urban areas we’ve examined.” She said her research team also is working on a study to examine whether there is evidence of some students’ being counseled out, or pushed out because of disciplinary issues, “to see if we can see any evidence of systematic behavior in that regard. And I don’t have those answers yet.”

Funding, Methodology

Sparks of Education Week advised journalists interested in venturing into charter school research to not only look at methodology but also who funded the study.

Raymond pushed back on that tip somewhat.

“How rigorous is the methodology? That is my first question,” she said. “Reporters need a basic review on what a good research methodology looks like. If the research design is good, you can’t really mess with the data. So I look at design before I look at the author.”

But she added, “You’ll be able to see any bias over a series of articles. Look for if they trend pro or con or neutral.”

Raymond said she has had to walk away from funders who wanted to set out a study’s conclusion in advance. “I vet my funders. I ask, ‘Are you going to be unhappy if I throw you under the bus?’”she said. “I’ve been sent to Siberia by some funders.”

CREDO has received financial support from numerous sources for its research, including the Michael & Susan Dell Foundation, the Robertson Foundation, and the Walton Family Foundation. The National Education Policy Center also has received financial support from various organizations, including the Atlantic Philanthropies, the Ford Foundation, and the two national teachers’ unions — the American Federation of Teachers and the National Education Association. (The Education Writers Association has received grants from the Ford Foundation, Michael & Susan Dell Foundation, and the Walton Family Foundation as well as event sponsorships from the AFT and NEA.)

One thing that both panelists and the moderator agreed on was how important it is for reporters to read entire reports, even when the page count climbs into the hundreds.

“What we have found is the executive summary and press release is where you are most likely to find spin,” said Welner.

The Research ‘Frontier’

Moving forward, Raymond said that the work of journalists is important to answering the big questions about charter schools. She pointed to work done by Danielle Dreilinger of the New Orleans Times-Picayune as a prime example of the importance of reporting (Dreilinger’s series on the prospects for students who graduated from KIPP charter schools in New Orleans was supported by an Education Writers Association Reporting Fellowship.)

“I can tell you what the KIPP data look like, but she’s telling a much different story on the ground,” Raymond said. “I think it’s the combination of those two things that actually represent the research frontier.”