Evidence for ESSA and the What Works Clearinghouse

HuffPost | Robert Slavin | January 12, 2107

In just a few weeks, we will launch Evidence for ESSA, a free web site designed to provide education leaders with information on programs that meet the evidence standards included in the Every Student Succeeds Act (ESSA). As most readers of this blog are aware, ESSA defines standards for strong, moderate, and promising levels of evidence, and it promotes the use of programs and practices that meet those standards.

One question I frequently get about Evidence for ESSA is how it is similar to and different from the What Works Clearinghouse (WWC), the federal service that reviews research on education programs and makes its findings available online. In making Evidence for ESSA, my colleagues and I have assumed that the WWC will continue to exist and do what it has always done. We see Evidence for ESSA as a supplement, not a competitor to the WWC. Evidence for ESSA will have a live link to the WWC for users who want more information. But the WWC was not designed to serve the ESSA evidence standards. Ruth Neild, the recently departed Acting Director of the Institute for Education Sciences (IES) (which oversees the WWC) announced at a November (2016) meeting that the WWC would not try to align itself with the ESSA evidence standards.

Evidence for ESSA, in contrast, is specifically aligned with the ESSA evidence standards. It follows most WWC procedures and standards, using similar or identical methods for searching the literature for potentially qualifying studies, computing effect sizes, averaging effect sizes across studies, and so on.

However, the purpose of the ESSA evidence standards is different from that of the WWC, and Evidence for ESSA is correspondingly different. There are four big differences that have to be taken into account. First, ESSA evidence standards are written for superintendents, principals, teachers and parents, not for experts in research design. The WWC has vast information in it, and my colleagues and I depend on it and use it constantly. But not everyone has the time or inclination to navigate the WWC.

Second, ESSA evidence standards require only a single study with a positive outcome for membership in any given category. For example, to get into the “Strong” category, a program can have just one randomized study that found a significant positive effect, even if there were ten more that found zero impact (although U.S. Department of Education guidance does suggest that one significant negative finding can cancel out a positive one). Personally, I do not like the one-study rule, but that’s the law. The law does specify that studies must be well-designed and well-implemented, and this allows and even compels us, within the law, to make sure that weak or flawed studies are not accepted as the one study qualifying a program for a given category. More on this in a moment.

Third, ESSA defines three levels of evidence: strong, moderate, and promising. Strong and moderate correspond, roughly, to the WWC “meets standards without reservations” (strong) and the “meets . . . with reservations” (moderate) categories, respectively. But WWC does not have anything corresponding to a “promising” category, so users of the WWC seeking all qualifying programs in a given area under ESSA would miss this crucial category. (There is also a fourth category under ESSA which is sometimes referred to as “evidence-building and under evaluation,” but this category is not well-defined enough to allow current research to be assigned to it.)

Finally, there has been an enormous amount of high-quality research appearing in recent years, and educators seeking proven programs want the very latest information. Recent investments by IES and by the Investing in Innovation (i3) program, in particular, are producing a flood of large, randomized evaluations of a broad range of programs for grades Pre-K to 12. More will be appearing in coming years. Decision makers will want and need up-to-date information on programs that exist today.

Evidence for ESSA was designed with the help of a distinguished technical working group to make common-sense adaptions to satisfy the requirements of the law. In so doing, we needed to introduce a number of technical enhancements to WWC structures and procedures.

Ease of Use and Interpretation

Evidence for ESSA will be very, very easy to use. From the home page, two clicks will get you to a display of all programs in a given area (e.g., programs for struggling elementary readers, or whole-class secondary math programs). The programs will be listed and color-coded by the ESSA level of evidence they meet, and within those categories, ranked by a combination of effect size, number and quality of studies, and overall sample size.

A third click will take you to a program page describing and giving additional practical details on a particular program.

Three clicks. Done.

Of course, there will be many additional options. You will be able to filter programs by urban/rural, for example, or according to groups studied, or according to program features. You will also see references to the individual studies that caused a program to qualify for an ESSA category.

You will be able to spend as much time on the site as you like, and there will be lots of information if you want it, including answers to “Frequently Asked Questions” that go into as much depth as you desire, including listing our entire “Standards and Procedures” manual. But most users will get where they need to go in three completely intuitive clicks, and can then root around to their hearts’ content.

Focus on High-Quality, Current Studies

We added a few additional requirements on top of the WWC standards to ensure that studies that qualify programs for ESSA categories are meaningful and important to educators. First, we excluded programs that are no longer in active dissemination. Second, we eliminated measures made by researchers, and those that are measures of minor skills or skills taught at other grade levels (such as phonics tests in secondary school). Third, the WWC has a loophole, counting as “meeting standards without reservations” studies that have major flaws but have effect sizes of at least +0.25. We eliminated such studies, which removed studies with sample sizes as small as 14.

Definition of Promising

The WWC does not have a rating corresponding to ESSA’s “Promising” category. Within the confines of the law, we established parameters to put “Promising” into practice. Our parameters include high-quality correlational studies, as well as studies that meet all other inclusion criteria and have statistically significant outcomes at the student level, but not enough clusters (schools, teachers) to find significant outcomes at the cluster level.

Rapid Inclusion

Evidence for ESSA will be updated regularly and quickly. Our commitment is to include qualifying studies brought to our attention on the website within two weeks.

Evidence for ESSA and the WWC will exist together, offering educators two complementary approaches to information on effective programs and practices. Over time, we will learn how to maximize the benefits of both facilities and how to coordinate them to make them as useful as possible for all of the audiences we serve. But we have to do this now so that the evidence provisions of ESSA will be meaningful rather than an exercise in minimalist compliance. It may be a long time before our field will have as good an opportunity to put evidence of effectiveness at the core of education policy and practice.

Robert Slavin is Director of the Center for Research and Reform in Education at Johns Hopkins University School of Education

This blog is sponsored by the Laura and John Arnold Foundation