Research & the ESSA Pt 2: The four-tiered problem

Last month, I discussed my concern that the ESSA’s expansion of the states’ role in program evaluation research threatens to overwhelm state departments of education (DOEs). I suspect that, given their limited capacity, most DOEs will not often choose to undertake their own program evaluation. Instead, they will outsource efficacy research to independent firms or rely on studies vendors provide on their own product.
 
Outsourcing and the Four-Tiered Problem
 
Potential outsourcing greatly increases the likelihood that the ESSA’s new four-tiered system for evaluating research quality will prove problematic. “Strong” evidence comes from randomized controlled trials, which are and should be the gold standard. “Moderate” evidence comes from data collected from thousands of students analyzed with multivariate statistical methods. The law may overstate the difference in evidence quality between the two tiers, but that’s a minor technical quibble.
 
Tiers 3 and 4 are more worrisome. “Promising” evidence comes from correlational studies that employ controls for selection bias. Every reader who has taken an introductory statistics course should be screaming “CORRELATION DOES NOT EQUAL CAUSATION” at the top of their lungs.  If selection bias is the only characteristic controlled, states run the risk of papering over a host of characteristics relevant to program efficacy. Does the “promising” tier also require controls for student socioeconomic, previous learning, and family characteristics, as it should? One of the federal DOE’s first tasks should be to carefully define the specifics of what the “promising” standard requires.
 
Whatever faults the “promising” tier has are overshadowed by those of the fourth tier. This tier requires only that “a state or provider can show that a program’s rationale is based on high-quality research.” Only George Orwell could appreciate a tier of “evidence” that requires nothing but a good argument. Any unscrupulous writer can link any product to a well-established base of research. Indeed, try finding a program on the market that does not say its theory of change is based on research. The role of research is to provide actual, empirical evidence on whether a product translates its theory of change into actual results for children.
 
Why It Matters
 
The capacity limits of state DOEs and the relaxed rigor of the bottom two tiers may mean that the ESSA diminish the quality of research that informs US education policy and classroom practice. Overburdened DOEs will rely on evidence that providers generate themselves. Focused on the bottom line, these providers will seek research that maximizes the likelihood of positive outcomes for their program, and the correlations sanctioned in the “promising” tier seem to invitefalse positives. Companies may opt to skip research entirely and rely on a skillfully written white paper that outlines a theory of change over actual empirical evidence of effectiveness. The burden to interpret evidence quality will still fall to DOEs, who still will not have the capacity to discriminate between competing studies that all claim to meet the ESSA's evidence quality standards. 
 
I hope I am wrong. 
Elizabeth Fuqua