back to list of Papers
back to list of Papers
Jane Butler Kahle
Condit Professor of Science Education
Miami University
420 McGuffey Hall
Oxford, OH 45056
513-529-1686 Phone
513-529-2110 FAX
kahlejb@muohio.edu
The preparation of this paper was sponsored in part by the National Science Foundation, Grant # OSR-92500 (J. B. Kahle and K. G. Wilson, Co-Principal Investigators). The opinions expressed are those of the author and do not necessarily reflect the position of NSF.
Introduction
Although much has been written about the nature and policies of systemic reform by policy makers, those papers provide only a partial vision of systemic reform--one from the outside in. Less has been written from the field; i.e., from those who are actively trying to promulgate reform either in the classroom or at the state level. Even less has been written about changes in teacher practices and in student learning-- yet, without those changes the reforms eventually will wither and fade away.
The encompassing nature of systemic reform provides critical roles for national and state leaders, for professional groups, and for individual teachers. Experience with several systemic initiatives in one state, Ohio, forms the basis for the following discussion of the challenges to reform and the changes needed for success. 1
My discussion focuses on the challenges faced by one state's efforts, on evidence of changes in teaching practice and in student learning, and on the meanings that may be drawn from that evidence. Specifically, Ohio's reform was characterized by the following parameters. It:
What Are the Challenges?
The challenges describe five aspects of systemic reform that must be addressed and aligned. Because each one poses risks to a part of the education community, varying levels of success have been reached. However, the Lessons Learned in attempting to meet the challenges provide insights and directions for the future.
Challenge One: Sustained professional development of a validated model can produce a culture shift in participants, but it is costly and time intensive. Given a state cohort of over 7,500 teachers per grade level, there is neither the human nor the financial resources to reach more than a small fraction of the target audience within a five year period. 2
Lesson Learned: Not only are human and financial resources limiting, but the pool of teachers who can, or will, undertake sustained professional development is limited. Teachers, who may be characterized as needing professional development the most-- ones in poor schools, ones with general licenses, ones with few courses in science or mathematics, ones teaching out of certification areas, ones who are disenchanted or disenfranchised--do not readily volunteer for a rigorous summer of mathematics or science. Rather, they must be reached in their communities and schools, the academic program must be at the level at which they teach science and/or mathematics, and the materials used must be directly applicable in their classrooms and with their students.
To meet this challenge, research validated curriculum were identified. Next, teachers, who had had at least one year of professional development, offered local, 40-hour, workshops for their peers. Districts supported the teacher-instructors, and often required all science or mathematics teachers at the targeted grade to attend. Obstensively, the workshops were to help teachers learn to use standards-based curricula; in reality, much mathematics and science was learned.
Challenge Two: Any reform has a limited and unique function. Although it must offer resources that are not available during its lifetime, those resources eventually must be assimilated into the ongoing educational system.
Lesson Learned: At the beginning of the reform, both the systemic initiative and the Ohio Department of Education divided the state into eight professional development regions. Two sets of centers were established which tested two different paradigms for professional development. The systemic initiative insisted upon regional collaboration before identifying and supporting its regional centers, which, then, delivered the six-week institutes. They were taught by regional academic leaders (both outstanding teachers and Ph.D. scientists and mathematicians) who were available during the school year to assist teachers in their classes, to work on curriculum teams, and to provide local workshops. The systemic initiative's model was successful beyond anyone's expectations. The Department's less costly model involved short-term, usually three-day, workshops with limited classroom follow-up. The teaching staff rotated and many were imported for a few days work. Furthermore, the selection process for the center was competitive, resulting in antagonism between the units funded (districts, colleges, county offices) and those passed over.
It was obvious that the two regional units needed to merge, and merger has occurred. The mergers have been slow and fraught with difficulties; for example, how to retain the sustained professional development model within the merged regional center. However, systemic reform is about taking chances, about building consensus, and about moving ahead with the results. In a large, complex state, regionalization is needed, and it is better to have one unit than none, and two units are not sustainable. Furthermore, the merged center is fully assimilated into the state's educational system.
Challenge Three: Systemic initiatives have underestimated the difficulty of getting mathematicians and scientists - at all levels - to work together, the difficulty of shifting university faculty from teaching by lecture to inquiry, as well as the difficulty in communicating between campuses and across disciplines. In Ohio, two separate groups of mathematicians developed mathematics by inquiry courses, because that process was more efficient in terms of time than the collaboration necessary to develop one course. Later, when both middle school mathematics and science teachers were in institutes on the same campuses (often in adjacent classrooms), both groups actively resisted working on integrated units or comparing strategies across disciplines.
Lesson Learned: It takes time and effort to encourage collaboration. Yet, without it, a reform is only pockets of change, not systemic. Therefore, the systemic initiative instigated collaborations with Ohio's three Urban Systemic Initiatives and the Appalachian Rural Systemic Initiative. Further, it identified collaborative relationships with Ohio's Mathematics and Science Coalition, the Parent Teachers Association, and many regional and local businesses and foundations.
Challenge Four: Although Ohio's systemic initiative focused on individual (or groups) of teachers, a school is a more viable unit of change. Teachers need a support system for the reforms they are initiating in their classrooms. A school focus also produces the visibility to attract external, market-driven resources that may continue the reform after the funding period. Changes in a school's science program or science department--with documented improvement in results--is a phenomenon that may be quickly communicated to parents and policy makers.
Lesson Learned: As part of the assessment, described below, brief site visits were made to 12 schools, primarily in poor urban or rural areas. Both the observational and questionnaire data gathered suggested that there were greater changes in learning environments, in teaching practices, and in student outcomes in schools that had a critical mass of reform teachers (up to one third of the mathematics/science faculty), compared to schools with only a few, or an isolated, teacher with the sustained professional development. A supportive group of teachers is especially important because of the high mobility of principals.3
Systemic or standards-based reform requires a critical and self-sustaining mass of teachers in a school. The solution was twofold: first, districts were requested to require or provide incentives so that all appropriate teachers would be involved in local professional development; and, second, the intensity and depth of the professional experiences were moderated with caution and some trepidation. (See Challenge One).
Challenge Five: All initiatives that are systemic in nature have important research/development and dissemination/support roles. Well-researched and validated professional development packages, such as Physics by Inquiry (McDermott, Shafer, & Rosenquist, 1996) have sustainability independent of the instructors. Such packages can be assimilated very quickly into existing delivery systems, such as a state's regional centers. Where such packages do not exist (or lack research validation), the systemic initiative must take the professional development packages through carefully controlled field tests and refinement activities in order to document their value and sustainability through research studies.
Lesson Learned: There is neither time nor money to do it all. Ohio has learned two lessons. First, find and use the expertise of others and, second, assess progress and outcomes in order to refine and improve your efforts. As mentioned earlier, in the fourth year of the reform, the systemic initiative undertook a major study to describe the landscape of science and mathematics education in state. The intent of the assessment was to tell the reform story in terms of changes in learning environments, in teaching practices, and in student learning. Because of the focus on equity, the schools selected to gather student achievement data were in poor, urban or poor, rural districts.
How well the systemic initiative met its challenges and what types of changes were evident are described next.
What are the Outcomes?
Four years into the reform, a comprehensive assessment of learning environments; teaching practice; principal, teacher, and student attitudes as well as student learning was undertaken. The study consisted of two levels and involved the collection of both quantitative and qualitative data. Level one consisted of a random sample of 126 schools, drawn from all the schools in the state that had at least one teacher who had completed the initiative's professional training. At level one, principals and all teachers who taught either science or mathematics (grades 5 through 9) completed questionnaires concerning classroom instruction, administrative support, and parent influence as well as on issues of school change.
Level two consisted of brief site visits to 12 (from the original random sample of 126) selected schools. In those schools, students and parents also completed questionnaires, students completed achievement tests, and principals, teachers, and randomly selected students and parents were interviewed. At each site school, a teacher who had had the sustained professional development (reform teacher) was matched with a teacher who had not had that experience (non-reform teacher). In addition, a randomly selected class of the reform teacher was matched with a comparable class of the matched, non-reform teacher. Changes were identified by comparing the responses of these matched groups of teachers and students.
Using 1990 and 1992 public release items from the National Assessment of Educational Achievement (NAEP), science and mathematics achievement measures were developed by teams of faculty, teachers, and regional leaders. Test items focused on process, not product, because the reform's goals were to increase conceptual understanding as well as skills needed to interpret and use scientific and mathematical information. In Miller's (1996) discussion of barriers to systemic reform, he notes that there is often a disconnect between the practical paradigm of reform (focused on process) and the technical paradigm of education (focused on product). This disconnect was avoided by developing new achievement measures.
When possible the questionnaires for principals, teachers, students, and parents contained the same questions--phrased appropriately. That strategy allowed comparison of responses across groups. For example, did both teachers and students respond that manipulatives were used at least once a week? The results indicate that students responded similarly to their teachers concerning instruction in reform and non-reform classes. That is, students in reform classes significantly more often: talked with each other about the subject, had to support their claims, and were encouraged to ask questions. Interestingly, significantly more students and teachers in reform classes reported that their principal had learned to accept classroom noise.
When the ways in which students learn were examined, interesting and significant differences were found between reform and non-reform classes. Students in classes taught by reform teachers significantly more often wrote about how they solve problems, solved problems in small groups, and used hands-on manipulatives. Those strategies are recommended both by the NSES, National Science Education Standards, (National Research Council, 1996) and by the research literature concerning strategies to improve the participation, attitudes, and achievement levels of girls and minority students. One of the six-week content courses incorporated the computer as a learning tool; another heavily used graphing calculators. Further, both the NCTM and NSES standards argue for the incorporation of technology into science and mathematics lessons (National Council of Teachers of Mathematics, 1989; NRC, 1996). However, neither practice with, nor information about, the efficacy of technology in promoting learning affected the use of calculators or computers. The lack of appropriate equipment and software remains a major challenge to implementing the reform agenda.
Briefly, there were significant differences in science achievement, as measured on the Discovery Inquiry Tests, in favor of reform classes. It is important to note that it was a low-stakes test (grades were not affected) and that it focused on a student's ability to interpret information and on conceptual, not factual, understanding. Because of the reform's focus on equity, the results also were examined for any sex, race, and/or group (reform versus non-reform) differences. Those analyses revealed interesting patterns of achievement; for example, minority students (in this case, African American) in reform classes scored significantly higher than their peers in non-reform classes. In fact, African American seventh and eighth graders in reform classes scored as well as White students in non-reform classes. Because the data were collected from reform and non-reform students taking the same type of class (e.g. general science, introductory life science, etc.) with a "match" teacher in the same school, economic differences were not a major factor.
Further, in science classes taught by teachers involved in the systemic reform, both African American and White girls scored higher than did the males in their racial group. In reform classes, White girls, which other research has shown to be the group most socialized away from science (Campbell, 1991; Campbell & Connoly, 1987; Kahle & Damnjanovic, 1996), scored higher than White boys or African American girls or boys. When science test scores were subdivided into physical science and life science items, girls in classes taught by the reform teachers scored higher than boys on the physical science items. These findings contrast with those of others. Using a large national data set (NELS: 88), gender differences have been found in achievement in physical science, but not in biology (Burkham, Lee, & Smerdon, 1995). These results may be the first time that a gender achievement gap in physical science, favoring girls, has been reported. Clearly, the type of teaching observed and recorded in responses to student and teacher questionnaires--more use of manipulatives, more time to talk about science, more opportunities to write about science, increased use of cooperative learning groups--has affected achievement, particularly for students who have been under-represented in science.
A Further Challenge
Once the data were collected and analyzed, the challenge was to distribute the findings widely in an accessible way. Over 10,000 copies of a small, easy-to-read publication, the Pocket Panorama (Kahle & Rogg, 1995), have been distributed across the state and nation. Because most state departments of education do not have either the time nor the expertise to perform large-scale research and dissemination activities, documentation, validation, and dissemination of change and of best practice provide unique and important roles for systemic initiatives. Such activities are critical for the public's understanding and acceptance of systemic reform. Indeed, research and dissemination may be the key roles for externally funded reform initiatives within a state. The final lesson learned is that research or assessment without dissemination benefits only those who are already involved in the reform. Dissemination of findings in practical and easy-to-use ways informs others of the initiative's success and invites them to become active participants in it. Further, assessment, coupled with wide- spread dissemination, provides the basis for successful reform strategies to become sustained through the existing educational system.
All parts of a reform must be addressed and work together if the results are to be systemic. The challenges in one state led to alterations in its reform strategies; those alterations, in turn, led to wider participation and acceptance of the reform. Although the changes described in the assessment cannot be directly attributed to the professional development and support strategies that were part of the reform, the findings suggest that improved learning is associated with improved practice that is initiated through sustained professional development.
Notes
1 - Ohio was one of the first ten states to receive National Science Foundation funding for a Statewide Systemic Initiative. Further, Ohio's three cities that were eligible for Urban Systemic Initiative funds, Cincinnati, Cleveland, and Columbus, have been awarded grants. Ohio also has five counties in the Appalachian Rural Systemic Initiative.
2 - NSF systemic initiative awards are for up to five years, although contracts are renewed annually. Three of the first cohort of states were terminated during the five- year cycle.
3 - In Ohio, over 50% of principals are in that position in a particular school for four years or less.
References
Burkham, D. T., Lee, V. E., & Smerdon, B. A. (1995). Gender and science learning early in high school: Subject matter and laboratory experiences. Unpublished paper, University of Michigan at Ann Arbor.
Campbell, J. R. (1991). The roots of gender inequity in technical areas. Journal of Research in Science Teaching, 28, 251-264.
Campbell, J. R., & Connoly, C. (1987). Deciphering the effects of socialization. Journal of Educational Equity and Leadership, 7(3), 208-222.
Kahle, J. B., & Rogg, S. R. (1996). A pocket panorama of the landscape study, 1995. Oxford, OH: Miami University.
Kahle, J. B., & Damnjanovic, A. (1996, April). Effects of inquiry teaching on the achievement levels of urban middle school science students by race and sex. A paper presentation at the annual meeting of the American Association for the Advancement of Science, Baltimore, MD.
McDermott, L. C., Shafer, P. S., & Rosenquist, M. L. (1996). Physics by inquiry. New York, NY: John Wiley and Sons.
Miller, K. W. (1996). Paradigmatic school philosophies as barriers to school reform. Science Educator, 5(1), 1-6.
National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.
National Research Council. (1996). National science education standards. Washington, DC: National Academy Press.