back to list of Papers

Prepared for Bernice Anderson of the National Science Foundation

SSI Strategies for Reform: Preliminary Findings
from the Evaluation of NSF's SSI Program

Andrew A. Zucker, SRI International
Patrick M. Shields, SRI International
Co-Directors, National Evaluation of the SSI Program
with the assistance of

Nancy Adelman, SRI International
Rebecca Carver, Stanford University School of Education
Thomas B. Corcoran, Consortium for Policy Research in Education
David Goldstein, Policy Studies Associates
Margaret Goertz, Consortium for Policy Research in Education
Daniel Humphrey, SRI International
Katrina G. Laguarda, Policy Studies Associates
Camille Marder, SRI International
Barbara Matson, SRI International
Steven Schneider, Woodside Research Associates
Choya Wilson, SRI International

February 1997

SRI International and its partners 1 are in the final year of a five-year evaluation of the National Science Foundation's Statewide Systemic Initiatives (SSI) Program. In preparation for writing a report on state strategies for systemic reform, the evaluation team has spent time examining every SSI's approaches to systemic reform. Preliminary findings about the effectiveness of different strategies are reported here.

Identifying SSI Strategies

In an earlier report 2 we identified eight strategies that the SSIs have used to carry out reform. In this paper, the strategies are grouped under two headings: strategies that focus principally on teachers, classrooms, and schools, and strategies that focus on districts, regions, and states. (See Exhibit 1.) This is a useful heuristic, even though the strategies often share some elements of both approaches. (We note that categorizing the strategies under these two headings is similar to what others have done in referring to "bottom up" and "top down" approaches to reform.)

Exhibit 1: Eight Strategies Used by the SSIs Strategies focused on teachers, classrooms, and schools: 1. Supporting teacher professional development 2. Developing, disseminating, or adopting instructional materials 3. Supporting model schools Strategies focused on districts, regions, and states: 4. Aligning state policy 5. Creating an infrastructure for capacity building 6. Funding local systemic initiatives 7. Reforming higher education and the preparation of teachers 8. Mobilizing public and professional opinion

Typically, an SSI will rely on one or more primary strategies and several secondary strategies as its participants undertake systemic reform in mathematics and science education. The evaluation team used a variety of documents about each SSI, as well as knowledge gathered during site visits, to make decisions about what the primary and secondary strategies are in each SSI, and these are shown in Exhibit 2. In some cases, SSI strategies have evolved or may even have changed dramatically, making it more difficult to assign strategies with certainty. Still, it seems likely that the SSI Principal Investigators would agree with the great majority of the decisions (nearly 100 of them) that have been made by the evaluation team.

Exhibit 2: THE SSIs' STRATEGIES FOR PROMOTING SYSTEMIC REFORM

States	Aligning State Policy	Supporting Teacher Professional Development	Creating an Infrastructure for Capacity Building	Developing & Disseminating New Content & Materials	Funding Local Systemic Initiatives	Supporting Model Schools	Reforming Higher Education & the Preparation of Teachers	Mobilizing Public & Professional Opinion
AR	S	P	P				S	S
CA		P	S
CO			S		P		S	S
CT			P		P		S	S
DE	S	P				P
FL	S	P				P	S	S
GA	S						P
KY		P	P	S		P	S	S
LA	S	P	S		S		S	S
MA	S	P	P		P		S
ME	S	P	S		P		S
MI	S		P		P		S
MT	S	S		P				S
NC		P	S
NE		P	P	S
NJ		S	P		P
NM	P	P	P	S			S	S
NY						P
OH		P	P
PR	S	P	P	P		P	S
SC	P	P	P					P
SD					P
TX	P		P				S
VA		P					P	S
VT	S	P	S	S	S			S

Note: P = Primary. S = Secondary. "Primary" means that a strategy constitutes the core thrust (or one of a few) of the SSI. "Secondary" means that it is a key portion of the strategy, but not central.

It is important to note that Exhibit 2 only reflects strategies that are directly part of the SSI. In other words, they are part of the "value added" by the systemic initiative. Thus, for example, although California clearly invested heavily in policy alignment, that largely occurred prior to the SSI and was not supported by SSI funds, and so Aligning State Policy is not shown as an SSI strategy in California. (Case studies of 12 SSIs that are being distributed as part of the evaluation will reflect the entire context in which each SSI operates, including other key strategies for education reform being used in those states apart from the SSI.)

Quality Ratings

In order to make judgments about the effectiveness of different reform strategies, the team identified a set of six common criteria to use in considering the implementation of each strategy. Each criterion was defined in detail for each of the eight strategies (see Appendix A). The six common criteria are:

1. Quality: how well the strategy has been executed (e.g., the quality of instructional materials);
2. Access and Inclusion: the degree to which the SSI-sponsored efforts have been open to the full state community (e.g., access of rural teachers to professional development);
3. Scale: the proportion of the total potential target audience reached and the strength of plans to reach the entire state;
4. Sustainability: the strength of the SSI's strategy for maintaining the reform effort after NSF funding ends;
5. Impact: the impact on the target audience (e.g., the degree to which teachers have improved their practice);
6. Theory of Change: the strength of the theory of change guiding the SSI strategy (e.g., support provided through SSI infrastructure reflects full needs of participating districts and schools).

The ratings were difficult to do, for a variety of reasons:

• First, unlike the case of the SSI mid-point reviews, which involved a fair-sized panel, for each SSI these ratings have been mainly the responsibility of a single individual. Although the team did try to "calibrate" the ratings, the reliability of specific ratings would have been significantly increased if there were multiple, independent raters.
• Second, the team¹s knowledge of the "non-case study" states is weaker than its knowledge of the case study states; thus, some of the SSIs were harder to rate than others.
• Third, it was more difficult to rate SSIs on some of the strategies than on others. For example, there is little hard information in most cases about the effectiveness of SSI strategies to mobilize public and professional opinion. As a result, ratings on that SSI strategy are more difficult to make with accuracy than on most of the other strategies.

I. Strategies Focused on Teachers, Classrooms, and Schools

SUPPORTING TEACHER PROFESSIONAL DEVELOPMENT.

Professional development is important if reform is to succeed. It is the most common SSI strategy (a high priority in 18 of 25 SSIs), and in our view the majority of the SSIs using this strategy can be rated as strong or very strong in the way they conduct professional development. Delivering high-quality professional development is something that we as a nation know how to do­at least for those teachers who are interested and when there are sufficient funds. The SSIs used three general approaches to professional development: local human resource development; local system capacity building; and state system capacity building. The SSIs typically used a mixed professional development strategy, and at least 9 employed all three approaches. In spite of the strong efforts made by the SSIs, the professional development system, such as it is­including state, district, and school policies related to professional development­is in need of restructuring. As long as professional development systems in the states remain unchanged, the SSIs, and the states, will face difficult trade-offs between working with large numbers of teachers superficially or working with small numbers intensively.3

DEVELOPING, DISSEMINATING OR ADOPTING INSTRUCTIONAL MATERIALS. Instructional materials are basic to what happens in schools. Yet only 6 SSIs focus on the instructional materials used in mathematics and science classrooms as a major part of their strategy. Of those, we estimate that 4 can be rated strong or very strong in their efforts. In many of the SSIs, instructional materials are still a "weak link," especially in certain domains (e.g., high school science) High-quality materials need to be identified or developed and decisionmakers need to be well informed about them. Whether decisions are made at the state or the local level, we believe it would make sense for more SSIs, and more states, to adopt and/or disseminate information about existing high-quality curriculum materials.

SUPPORTING MODEL SCHOOLS. Just 5 SSIs use a model schools strategy, and we would rate only 2 as doing a strong or very strong job. The evaluation team continues to believe this is a high-risk strategy. It requires a very careful plan for dissemination and scaling up. If such a plan is not well-designed and well-implemented, the result is, at best, improvement in a handful of schools statewide. In the two states that we believe have strong model schools strategies (Delaware and Puerto Rico), that effort serves as part of a much broader systemic reform strategy.

II. Strategies Focused on Districts, Regions and States

ALIGNING STATE POLICY. State policy is difficult to change because it is driven by myriad political and resource issues beyond the control of the SSI. Only 3 SSIs target state policy as a primary strategy, although 11 others make it a secondary strategy to align the policy system with SSI goals, and in still other SSI states (e.g., California) policy alignment has been carried out under auspices other than the SSI. The majority of the SSIs that have targeted state policy have done a strong job or better at it. The SSIs that have done particularly well in their efforts to align state policy­we estimate there are 9 of them­typically have multiple connections with state policymakers, political "savvy," ongoing systemic reform efforts with which the SSI is integrated, and the good fortune not to be caught in fast-changing political tides.

CREATING AN INFRASTRUCTURE FOR CAPACITY BUILDING. Success for the SSIs is defined in part by what they will leave behind after the NSF funds disappear, and a new or improved infrastructure (e.g., new non-profit advocacy institutions or better regional education centers) would be one tangible legacy. Of the 18 SSIs emphasizing this strategy, we believe about half can be rated as strong or very strong. Doing a good job usually involves having a viable theory of how the infrastructure will help to change the system, and making good connections between existing and new components of the infrastructure.

FUNDING LOCAL SYSTEMIC INITIATIVES. Nine SSIs, mostly in local control states, chose to support local systemic initiatives. More than half of the nine (5) rated as strong or very strong. The key factors in building strong local initiatives are sufficient support for participants at the local level and some kind of quality control mechanism.

REFORMING HIGHER EDUCATION AND TEACHER PREPARATION. While 14 of the SSIs have adopted this strategy, none rates as very strong for reform in teacher preparation and just 4 rated as strong. Moreover, only 2 used this as a primary SSI strategy. Changing teacher preparation has been a challenge in part because of the difficulty of changing institutions of higher education, and in part because of the fact that relatively few of the SSI resources have been used for this purpose. Lessons on how to effectively change teacher preparation may more likely come from NSF¹s Collaboratives for Excellence in Teacher Preparation program than from the SSI program.

MOBILIZING PUBLIC AND PROFESSIONAL OPINION. Many surveys and studies provide evidence that public and professional opinion is critically important for education reform. Of the 10 SSIs that place a significant emphasis on mobilizing opinion, the evaluation team believes 2 of them are doing a strong job. Although most of the 10 are doing at least a fair job, the SSIs have had a difficult time developing powerful strategies for mobilizing opinion. One reason may be that is that it is difficult to find appropriate measures of success for public relations initiatives, so it can be hard to know if one is being successful or not.

APPENDIX A:

Criteria for Judging SSI Strategies

Criteria	Aligning State Policy	Supporting Teacher Professional Development	Creating an Infrastructure for Capacity Building	Developing, Disseminating, or Adopting Instructional Materials	Funding Local Systemic Initiative	Supporting Model Schools	Reforming Higher Education and Teacher Preparation	Mobilizing Public and Professional Opinion
Quality	Development of ambitious and clear standards for student learning that serve as the basis for alignment of key policies shaping the educational system.	Meets subject matter needs; research-based, sustained, adequate time and intensity, models active learning, provides for engagement with ideas and peers; teachers involved in design, designed for varying learners; meets needs of teachers and schools	Qualified personnel, content expertise; sufficient resources, intensity, and duration to have impact on resource users.	Instructional materials that reflect national (NCTM, NSES) standards for content and pedagogy	Sufficient resources, incentives, and support to ensure that local sites can progress with local systemic reform	Sufficient resources, incentives, and support to ensure schoolwide improvement	Clear standards for high-quality teacher education; sufficient resources, incentives, and support to ensure substantive improvements; focus on both content and pedagogy	Clear and well articulated vision of high-quality mathematics and science; high-quality public relations material distributed through appropriate channels to target audiences
Access/ Inclusion	N/A	Broad array of teachers have access	Access is offered to the full range of teachers and schools in the state	Access to materials is offered to full range of teachers and schools in the state	Local systemic initiatives represent range of districts in the state; opportunity for participation was open	Model schools represent range of schools in the state; opportunity for participation was open	Full range of IHEs in state have access to SSI activities or to lessons from SSI pilots	Target of public relations to many key groups within the state
Scale	Full range of policies (e.g., assessment, professional development, teacher credentialling, school accreditation) are aligned	Serves substantial proportion of teachers with clear strategy for reaching majority of teachers in state	Structure and resources are sufficient to reach all or most of the state	Explicit plan in place for getting materials into all classrooms	Explicit and realistic plans for transferring lessons from local systemic initiatives to other sites	Explicit and realistic plans for transferring lessons from model schools to other sites	Explicit and realistic plans for reaching all major IHEs	Explicit and realistic plans for reaching audiences within the entire state
Sustainability	Policies are codified or at least formalized; support for policies is broad	System and funding in place to continue to support high quality professional development	Sufficient resources--financial, personnel, material--are available to maintain infrastructure	Explicit and realistic plan (e.g., use of commercial publisher) for maintaining availability of materials	Explicit and realistic plans for continuing support of local systemic initiatives	Explicit and realistic plans for continuing support of model schools	Explicit and realistic plans for continuing public relations efforts
Impact	Clear evidence of substantive policy changes aligned with standards	Evidence that teachers have gained new knowledge, skills, and attitudes	Teachers, schools and districts are taking advantage of resources available through the infrastructure	Materials are being used in classrooms resulting in improved practice	Clear evidence of changed local systems in terms of aligned policies, quality of professional development and curricular materials, and public and professional support	Clear evidence of improved schools in terms of professional culture, explicit planning, and changed classroom practice	Improved undergraduate classes in mathematics, science and/or education; better prepared entering teachers	Evidence of increased support for SSIÕs vision of high quality mathematics and science among public and/or professionals
Theory of change	Explicit strategy exists, rooted in political context of the state, for policies to impact local practice; incentives for teachers to participate	Professional development includes explicit strategy to support teacher change beyond workshops, including addressing schoolwide change needs and contextual constraints	Support provided through the infrastructure is of sufficient duration and infrastructure fits the political and resource context of the state	Materials are accompanied by sufficient support for teachers so that they can impact classrooms; incentives to use materials	Quantity, quality, scope, and sequencing of support to local initiatives is explicitly tied to a realistic theory of what local systems need to improve mathematics and science for all students	Quantity, quality, scope, and sequencing of support to schools is explicitly tied to a realistic theory of what local systems need to improve mathematics and science for all students	Explicit strategy for addressing the incentive issues associated with changing higher education	Explicit strategy for changing public and professional opinion that takes into account preconceived notion of good mathematics and science

Notes

1 - Partners include the Consortium for Policy Research in Education (CPRE), Policy Studies Associates, Woodside Research Consortium, and the Council of Chief State School Officers.

2 - Zucker, A., Shields, P., Adelman, N., and Powell, J. (1995). Evaluation of NSFÕs Statewide Systemic Initiatives (SSI) Program, Second Year Report: Cross-Cutting Themes.

3 - Our report entitled The SSIs and Professional Development for Teachers is currently in draft form.

back to list of Papers