|
Why “What Works” Didn’t in L.A. Unified School District
A recent article in the Los Angeles Times reported that a $50 million reading improvement initiative featuring the Waterford Early Reading Program failed to improve reading and may even have hindered achievement for some students. The Waterford program is listed in the Metiri TSW Literacy database in the category, “What Works,” a designation that suggests strong evidence of effectiveness. Will the L.A. program evaluation cause us to modify our rating? Absolutely not! While Los Angeles Unified deserves kudos for conducting such a thorough and honest evaluation, in fact LAUSD’s implementation of Waterford provides a textbook study on how not to implement a technology-based intervention in your district.
High need and a good vision for ways to meet those needs are not enough; truly effective use of any teaching and learning tool requires careful planning, systemic implementation, and constant monitoring and adjusting.
Unfortunately, LAUSD seems to have stopped a little short of this (in some classrooms, according to the Times, they stopped before they even began!) And what happened as a result? The “solution” didn’t work for them.
Metiri regularly conducts seminars on implementing research-based learning technology solutions in schools. In them, we describe our recommendations for an important, eight-step process. These steps, and our analysis of the Waterford implementation in L.A. as reported by the Times, serve as a cautionary tale that districts should heed prior to implementing any of the “Solutions That Work.” The first steps are as follows:
Step 1: Visioning
Step 2: Identifying Learning Needs
Step 3: Reviewing Research and Best Practice
Step 4: Designing or Selecting an Intervention.
These may or may not have been attended to in the L.A. planning process. But it’s in the next set of steps - during the implementation phase – that we think (based on the L.A. Times report) major mistakes were made. These next steps break down as follows:
Step 5 – Ensure Readiness and System Support
Recent work done by the Metiri Group with the Milken Exchange on Education Technology, the North Central Educational Laboratory, and the State Educational Technology Director’s Association has consistently focused on the need for systems thinking during the technology integration process. Systemic elements of readiness and support that were specifically named as problems in the L.A. implementation include:
-Lack of facility readiness
A search of the LAUSD website finds references such as “Waterford computers are overloading school site electrical systems.” This suggests that the readiness of facilities in this district may not have been assessed prior to implementation.
-Lack of sufficient, effective professional development
The high level of non-implementation found by program evaluators at a date late in the initiative suggests that the associated professional development lacked the characteristic we find to be most essential if new skills and strategies are to be implemented in the classroom: a strategy for getting “behind the classroom door” to monitor the level and quality of implementation.
-Lack of alignment with existing curricula and instructional materials
The reading program that being implemented in all participating LAUSD schools while Waterford was being used was “Open Court.” Open Court is a highly structured program that provides teachers with scripted lessons from which they are encouraged not to vary. This is hardly a match for a software implementation that is designed for a flexible environment where students rotate through a limited number of computers during the instructional period. According to the Times article, a number of L.A. teachers reported being unable to use the Waterford program, since they were “forced to devote most of their mornings, and some afternoons, to these scripted lessons.”
Step 6 – Pilot and Assess Formatively
While some mention is made in LAUSD’s announcement of the purchase of Waterford that the purchase was made based on the experiences of teachers using the program in L.A. and other districts, there is no evidence that the district conducted a rigorous, controlled pilot prior to making the $50 million purchase. We consistently recommend to our clients that they conduct a formal, “scientifically-based” study to confirm the effect of any intervention being considered for wide-scale use. Such a study should verify the impact on local students through local teachers in local schools. Chicago Public Schools, for example, has commissioned a study comparing the impact of WiggleWorks, Earobics, and LeapTrack in Chicago classrooms with a set of control classrooms. This study is being undertaken prior to making large purchases of any of those interventions.
An additional benefit to such pilots is the formative data they provide. Such issues as infrastructure concerns, inadequate training, and non-implementation can be identified prior to wider implementation and strategies for dealing with these problems can be devised and tested, smoothing the path for the larger implementation.
Step 7 – Scale and Monitor
Wise scaling of an implementation seldom involves introducing the program in 2,235 classrooms in 244 schools simultaneously, as occurred in L.A.! Scaling should be done in a careful, scaffolded fashion that ensures just-in-time support and the “friendly accountability” of onsite resource people who can ensure that implementation is occurring. One of our favorite models for sensible systemic scaling was observed in the Montgomery County, Maryland school district. Montgomery County had created a vision for technology-supported learning named, “Global Access.” Global Access schools received technology, connectivity, and professional development that allowed them to participate in the program. Rather than roll the program out district-wide, they invited a limited number of schools into the program each year. During their first year in the program, participating schools received on-site technical support, professional development, and instructional support. In future years, support was lessened as schools became self-sustaining. This type of systemic scaling with scaffolded support for teachers, along with the “friendly accountability” of trainers and support staff in the classroom, ensures that teachers feel comfortable implementing a program and that concerns are dealt with quickly, keeping implementation schedules on track.
Step 8 – Conduct a Thorough Summative Evaluation
This step L.A. Unified seems to have accomplished…for better or for worse!
L.A. Unified’s experience confirms the necessity of the Eight Step process we have described above. Effective implementation of research-based learning technologies is not different than that of any other complex teaching and learning strategy: it requires careful planning, systemic implementation, and constant monitoring and adjusting during all phases of the rollout. To do otherwise is to court disaster – and thereby contribute to the popular misconception that technology doesn’t work.
 |
