back to list of Papers
back to list of Papers
by
Thomas B. Corcoran
CPRE
University of Pennsylvania
February, 1997
What is systemic reform?
Systemic reform as defined by scholars, policy makers, and funding agencies is an approach to school reform that views the policies and actions of government - school districts, states, the federal government - as the critical levers for improving the performance of public education. The central argument is that if governments set standards for student performance, and adopt aligned policies for curriculum, assessment, accountability, and governance, educators will alter their practices, and performance will improve. This reasoning distinguishes systemic reform from other reform strategies that rely on the creation of markets, the professionalization of teaching, or the work of volunteer networks of schools or educators as the critical forces for reform.
Proponents of systemic reform believe the performance problems of public education are primarily the result of low standards, incoherent and fragmented policies, and poor use of resources. Hence, they advocate higher standards and more robust and more coherent policies that send clearer and more compelling signals to educators, students, and parents about what is expected. The core set of policies addressed by systemic reform are often referred to as "drivers" or sometimes as the instructional guidance system. Typically they include content and performance standards, aligned assessments and curricular frameworks, an accountability system with rewards and sanctions, changes in teacher development that enable teachers to prepare students to meet the new standards, and decentralization. In some versions of this approach, schools performing well will be rewarded, and those that fail to make progress will be subject to a variety of sanctions. Advocates also propose reducing the regulations, bureaucracy, and policy incoherence that might impede reform and decentralizing decisions about how to best meet the standard (Cohen, 1995). In short, systemic reform seeks to make "the system" more rational, more coherent, more focused, more efficient. The approach is also sometimes called standards-based reform.
Systemic reform has obvious appeal. Its focus, comprehensiveness and specificity promise a more powerful strategy than the school by school approaches of the networks and a fairer, more orderly one than the markets envisioned by the choice advocates. Its logic is so compelling that its adherents often act as though it were a proven formula that could guarantee success. The only question of interest to the most committed enthusiasts often seems to be how to persuade reluctant state and local policy makers or educators to comply with the postulates of their theory.
In fact, systemic reform is a persuasive, but unproven, general theory of action. It is a general theory because it provides only a broad and rather abstract framework for policy design. It must be adapted to fit particular political contexts. As a consequence, the specifics of enacted theories of systemic reform vary from one setting to another. For example, consider the variations in the standards set by states and by localities. Or the variations in the content and form of the assessments used. These variations raise questions of interest to an evaluator. What are their consequences for practice and performance? To what degree can systemic reform be customized to a particular political setting and still be systemic?
I refer to the theory of systemic reform as unproven because as yet we have little empirical evidence to support the efficacy claims made by some of its advocates. We can draw some limited inferences from effects of the basic skills movement in the 1970s and 1980s and from the impact of similar approaches in other countries. And the early data from Kentucky are promising. But the empirical support for systemic reform remains thin. Some advocates seem to confuse evidence documenting the problems they feel are central (e.g., lack of standards, fragmentation, etc) with evidence of effectiveness. They are not the same thing. And little is known about the importance and relative efficacy of the various components of systemic reform. Are they equally essential? Is high stakes assessment needed to create incentives for improved performance? Are incentives needed for students? Is devolution of authority necessary to achieve the desired results? These are also questions of interest to evaluators.
The effects of systemic reform under varying conditions also need examination. Is fiscal equity a prerequisite condition for success? What are the educational and political effects of implementing standards and high stakes in situations with inequitable resource distributions? What are the consequences for schools with varying capacity to design and implement changes? What degree of fit is needed between teacher knowledge and skill and the standards?
I could go on. There are many unanswered questions. I was asked to reflect on the role of evaluation in systemic reform. Taking advantage of the broad scope of this charge, my comments will focus on the assumptions underlying "systemic reform" and on their implications for evaluation, on some other critical problems faced by those attempting to evaluate "systemic reform" and on some of the questions (in addition to those I have already raised) that such evaluation efforts ought to be addressing.
The Premises or Assumptions of Systemic Reform
The theory of systemic reform rests on some assumptions that should be carefully examined and tested. First, systemic reform seeks greater coherence, an alignment of policies, but the education system itself is fragmented by design - fifty states, fifteen thousand districts, countless other agencies impacting the schools - and this fragmentation is intended to permit variation. The agencies of government responsible for the schools are divided from each other by the federal structure and by the separation of powers. They are further divided by powerful traditions of local control and parental rights. On top of that within any given jurisdiction there are a variety of stakeholders each with their own views about standards, assessment, locus of authority, etc. What does coherence mean in this environment?
Systemic reform assumes that some consensus can be developed around standards, and that the resulting set of coherent policies will focus this fragmented system and improve its performance. From an evaluator's perspective these assumptions must be viewed as problematic. Previous research indicates that context is a dominant influence on the implementation and effectiveness of reforms. And the contexts in the states and school districts vary widely; state politics vary, their histories vary, their resources vary, and so on. Yet some contend that the key components of systemic reform must be essentially the same in every setting, including systemwide standards and assessments, and that these components will operate with similar effects under varying conditions. However, states with strong traditions of local control might find it difficult to adopt the recommended policies. Is it not possible to design a "systemic" approach that respects these traditions of local control? Could a systemic reform strategy that lacked one or more of these critical drivers be effective?
Second, advocates of systemic reform aim to change teaching. They speak of more coherent policies driving instruction. They assume that there is a strong linkage between policy and practice. However, research has found little evidence of such a linkage (Cohen and Spillane, 1992; Elmore, 1995). Moreover, while some advocates of systemic reform argue that they simply want to define the outcomes and that the means of achieving them might vary (hence the emphasis on devolution), others package specific notions of "best practice" together with their instructional guidance system so that the reform defines both the means and the ends. It is hardly surprising that this approach produces some resistance in a complex decentralized political system.
David Cohen (1994) has suggested that evaluators should be comparing the content and coherence of instructional guidance systems with what teachers, students, and parents comprehend to be its message. His point is that given the fragmented nature of the system along with the many competing messages about priorities coming from government, professional groups, reformers, and local communities, it would not be surprising if the messages were distorted. Certainly school districts and schools respond differentially to the same policy messages (Spillane, 1996). Evaluators should be examining the efficacy of the links between policy and practice.
Third, there is the question of the variation in schools' capacity to respond (Slavin , 1995). Systemic reformers aspire to raise the performance of all students and close gaps in achievement between advantaged and disadvantaged groups. Yet schools' capacity to respond to the policy signals or use the support that is provided varies. How does systemic reform affect schools at different levels of readiness? How do we engage teachers who resist changing their practice or participating in professional development? How effective are the intervention techniques being employed?
Fourth, the theory focuses on the system rather than students and seems to treat students as passive vessels with little or no responsibility for their own success. The system is seen as the problem, and if the standards and drivers are put in place, advocates contend the resulting concentration of energy will produce a more powerful pedagogy, higher student motivation, and improved performance. Shouldn't the response of students to these changes in policy and practice be viewed as problematic? What do we know about the link between educational policy and student behavior? How do different incentives affect the behavior of different groups of students? These are questions of importance to an evaluator of systemic reform.
Fifth, there is the problem of the standards. They are sometimes treated as a purely political question in systemic reform - what can we all agree to about what students should know and be able to do? The major concern seems to be that they are high enough. But as TIMSS has dramatically illustrated the developmental sequence of the content standards and their breadth and depth matters a great deal and shapes the outcomes. So there are empirical and technical questions to asked about the standards. Is more specificity better? Are inter- disciplinary approaches effective? Under what conditions? How many standards can be crowded into the curriculum? Who should set them? Does the locus of power matter?
How should we evaluate systemic reform?
To design evaluations of systemic reform initiatives, we must know something about the success criteria held by various stakeholders. Are the success criteria to be successful enactment of the policy instruments (e.g., the instructional guidance system), changes in classroom practice, changes in student performance, or all of the above? And with regard to performance, are we looking for improvement, getting all students to high standards, reducing gaps among significant groups, or all of the above? The answers to these questions seem to vary across audiences.
When federal policy makers talk about evaluating systemic reform, they often seem to be asking whether the government in question - state or district - has aligned its policies, put the drivers in place, provided the necessary supports, and is realizing the desired changes in practice and performance. The underlying assumption here is that systemic reform is a proven strategy and that we know how to do it, and therefore the only important question is "are they doing it right?" If the policy configuration is not as desired or the results are not as expected, the conclusion often is that they simply didn't get it. Such evaluations seldom ask questions about the robustness or costs of the strategy itself, whether some iterations of the strategy might be more effective than others, or how well the strategy works in different political and institutional contexts.
In my judgment this "checklist" approach to evaluation is a mistake. We should approach variations in strategies with an open mind, and examine their design, viability, costs, and impacts. Rather than defining systemic reform rigidly, we should respect the decentralized nature of our system and encourage and evaluate strategies that share a set of core ideas: challenging academic standards, coherence and focus, and systemwide change. We might learn that different policy mixes or sequences of action or distributions of authority work better than others under specific conditions.
Other Problems of Evaluation
Getting the Data. There are serious data problems that compound the problems of evaluating systemic reform. Data on classroom practice are typically collected through surveys because observations are too labor intensive to collect an adequate sample over space and time. Since the point of systemic reform is send clearer signals, teachers soon learn what the desired practices are, or at least what the code words are, and in high stakes environments may feel pressured to indicate compliance. Surveys may exaggerate the extent of the changes in practice.
There are worse problems with student performance data. Many states and districts do not assess all areas in which we seek to set standards or do not do so adequately. New measures are under development but are not yet trouble free. Performance measures are expensive and plagued with unresolved technical problems. This is also true of portfolios. The results of more conventional paper and pencil measures are not trusted by reformers who argue that they are not sensitive to the outcomes they care about, and furthermore, they undermine their efforts to alter practice. So what is an evaluator to do? We may be in the midst of a great transition in assessment but evaluators need valid, reliable, and stable measures to examine changes over time.
Timelines. What is a reasonable time frame in which to expect results? Policy makers and grantmakers have trouble waiting two or three years, but we know it takes more time than that to put the policies and supports in place and achieve broad scale changes in practice. Five years seems far too short to expect the broad scale changes reformers believe are needed to produce significant gains in achievement. Kentucky's policy makers have said it will take twenty years. Connecticut has a plan for a generation. Philadelphia's leaders talk about a decade. What is a reasonable time frame? How do capacity and previous experience affect time frames?
Attribution. Policy makers and funding agencies want to know that their strategy paid off. They want to take credit for results. How do we deal with attribution? When we get improved results, how can we be sure it was the standards or the" best" practices that were responsible? Maybe it was just the tests. Maybe it was the extra resources. Maybe it was simply a general cultural shift. How will we know what "caused" the improvements? Or when we do not get results, has systemic reform failed, or were there inadequate resources, poorly prepared teachers, too many external problems for schools to overcome? What interventions mattered? How will we know? We need well-designed, longitudinal cross-jurisdictional studies to answer these questions.
What do we need to know about systemic reform?
There is a great deal that we do not know, and unfortunately we are not getting answers from current evaluation efforts. Consider for a moment the following questions:
These are just some of the questions that we should be seeking answers to in order to advance systemic reform from the status of a promising general theory to a empirically grounded strategy for school reform.
References
Cohen, D. (1994). Evaluating systemic reform. A paper prepared for the Office of the Assistant Secretary for Planning and Evaluation, US Department of Education.
Cohen, D. (1995). What is the system in systemic reform? Educational Researcher, 24, 11- 17.
Cohen, D. and Spillane, J (1992). Policy and practice: The relations between governance and instruction. Review of Research in Education, 18.
Elmore, R. (1995). Structural reform in educational practice. Educational Researcher, 24, 23- 26.
Elmore, R. (1996). Getting to scale with good educational practice. Harvard Educational Review, 66, 1-26.
Slavin, R. E. (1995). Sands, bricks, and seeds: School change strategies and readiness for reform. Baltimore, MD: Johns Hopkins University, Center for Research on the Education of Students Placed at Risk.
Spillane, J. School districts matter: Local educational authorities and state instructional policy. Educational Policy, 10, 63-87.