There has been a lot of discussion on my blog (and in meetings) about choices we make in education, and that includes the type of interventions we use. One type of intervention used is for professional development for teachers (including both work outside of school and in-school work such as "coaching). Two recent review of the research on the effectiveness of such professional development (one on reading, one on math) unfortunately suggest that this approach has no impact on student achievement. I've posted the Executive Summaries of both reports below (as well as the links to the full reports for those who want the whole story).
Executive Summary: Impact of Two Professional Development Interventions on Early Reading Instruction and Achievement—NCEE 2008-4030, September 2008Professional development (PD) of teachers is viewed as a vital tool in school improvement efforts (Hill 2007). The importance of professional development (PD) for teachers is underscored in several major federal education initiatives, including the No Child Left Behind (NCLB) statute. For example, Title II of NCLB provided $585 million to states and districts for PD activities during the 2002-2003 school year alone in order to meet the goal of having a highly qualified teacher in every classroom (U.S. Department of Education, 2005). Two years later, Title II funding for PD remained at over $500 million (U.S. Department of Education 2007).
Are teachers receiving the PD that they need? A recent national study of state and local NCLB implementation indicated that 80 percent of elementary teachers reported participating in 24 hours of PD on reading instruction or less during the 2003–2004 school year and summer (U.S. Department of Education 2007). Reading and PD experts have raised a concern that this level of PD is not intensive enough to be effective, and that it does not focus enough on subject-matter knowledge (Cohen and Hill 2001; Fletcher and Lyon 1998; Foorman and Moats 2004; Garet, Porter, Desimone, Birman, and Yoon 2001).
To help states and districts make informed decisions about the PD they implement to improve reading instruction, the U.S. Department of Education commissioned the Early Reading PD Interventions Study to examine the impact of two research-based PD interventions for reading instruction: (1) a content-focused teacher institute series that began in the summer and continued through much of the school year (treatment A) and (2) the same institute series plus in-school coaching (treatment B). The study team consists of AIR, MDRC, and REDA International, Inc., who conducted the research activities, and Sopris West and the Consortium on Reading Excellence (CORE), who delivered the teacher and coach PD.
The Early Reading PD Interventions Study used an experimental design to test the effectiveness of the two PD interventions in improving the knowledge and practice of teachers and the reading achievement of their students in high-poverty schools. It focused specifically on second grade reading because (1) this is the earliest grade in which enough districts collect the standardized reading assessment data needed for the study; and (2) later grades involve supplementary (pull out) instruction, which was outside the scope of the study. The study was implemented in 90 schools in six districts (a total of 270 teachers), with equal numbers of schools randomly assigned in each district to treatment A, treatment B, or the control group, which participated only in the usual PD offered by the district. This design allowed the study team to determine the impact of each of the two PD interventions by comparing each treatment group’s outcomes with those of the control group, and also to determine the impact of the coaching above and beyond the institute series by comparing treatment group B with treatment group A.
This report describes the implementation of the PD interventions tested, and examines their impacts at the end of the year the PD was delivered. In addition, we investigate the possible lagged effect of the interventions, based on outcomes data collected the year after the PD interventions concluded.
The study produced the following results:
*
Although there were positive impacts on teacher’s knowledge of scientifically based reading instruction and on one of the three instructional practices promoted by the study PD, neither PD intervention resulted in significantly higher student test scores at the end of the one-year treatment. Teachers in schools that were randomly assigned to receive the study’s PD scored significantly higher on the teacher knowledge test than did teachers in control schools, with standardized mean difference effect sizes (hereafter referred to as “effect sizes”) of 0.37 for the institute series alone (treatment A) and 0.38 for the institute series plus coaching (treatment B). Teachers in both treatment A and treatment B used explicit instruction to a significantly greater extent during their reading instruction blocks than teachers in control schools (effect size of 0.33 for treatment A and 0.53 for treatment B). However, there were no statistically significant differences in achievement between students in the treatment and control schools.
*
The added effect of the coaching intervention on teacher practices in the implementation year was not statistically significant. The effect sizes for the added impact of coaching were 0.21 for using explicit instruction, 0.17 for encouraging independent student activity, and 0.03 for differentiating instruction, but these effects may be due to chance.
*
There were no statistically significant impacts on measured teacher or student outcomes in the year following the treatment.Executive Summary: Middle School Mathematics Professional Development Impact StudyThis report presents interim results from the Middle School Mathematics Professional Development Impact Study, which is sponsored by the Institute of Education Sciences (IES). The report presents results immediately following one year of the study’s professional development. A future report will present results following two years of professional development.
Student achievement in mathematics has been a focal concern in the United States for many years. The National Research Council’s 2001 report and the recent report of the National Mathematics Advisory Panel (2008) both called attention to student achievement in mathematics, and both called for all students to learn algebra by the end of eighth grade. Reports have argued, further, that achieving this goal requires that students first successfully learn several topics in rational numbers — fractions, decimals, ratio, rate, proportion, and percent. These topics are typically covered in grades 4 through 7, yet many students continue to struggle with them beyond the seventh grade. The National Mathematics Advisory Panel wrote that “difficulty with fractions (including decimals and percent) is pervasive and is a major obstacle to further progress in mathematics, including algebra” (p. xix). The panel also specified that by the end of seventh grade, “students should be able to solve problems involving percent, ratio, and rate, and extend this work to proportionality” (p. 20).
The U.S. Department of Education’s National Center for Educational Evaluation and Regional Assistance (NCEE) — within the Institute of Education Sciences — initiated the Middle School Mathematics Professional Development Impact Study to test the impact of a professional development (PD) program for teachers that was designed to address the problem of low student achievement in topics in rational numbers. The study focuses on seventh grade, the culminating year for teaching those topics. The study is being conducted by the American Institutes for Research (AIR) and MDRC together with their evaluation partners REDA International and Westat.
Currently, through the Elementary and Secondary Education Act, the federal government provides significant resources for PD, but little rigorous evidence is available on the impact of PD on teacher and student outcomes. Hundreds of studies have addressed the topic of teacher learning and PD (for reviews, see Borko 2004; Clewell, Campbell, and Perlman 2004; Kennedy 1998; Richardson and Placier 2001; Supovitz 2001; Yoon, Duncan, Lee, Scarloss, and Shapley 2007). The most recent review of studies of the impact of teacher PD on student achievement revealed a total of nine studies that have rigorous designs — randomized control trials (RCTs) or certain quasi-experimental designs (QEDs) — that allow causal inferences to be made (Yoon et al. 2007). Four of the nine studies focused on the effect of a PD program on mathematics achievement, and none focused on mathematics at the middle school level.
The Middle School Mathematics PD Impact Study is the first rigorous test of the impact of a PD program focused on teachers of middle school mathematics. Within 12 participating school districts, the study randomly assigned 77 mid- and high-poverty schools to treatment and control conditions and collected outcome data on teachers and students. The PD was delivered by two provider organizations, each of which served the treatment schools in six of the 12 participating districts. Seventh-grade teachers in the treatment schools had the opportunity to receive the PD program offered by the study and could also continue to participate in the PD activities that they would have received in the absence of the study. Seventh-grade teachers in the control schools received only the PD that they would have received in the absence of the study.
The study has three central research questions: - What impact did the PD program provided in this study have on teacher knowledge of rational number topics?
- What impact did the PD program provided in this study have on teacher instructional practices?
- What impact did the PD program provided in this study have on student achievement in rational number topics?
The study produced the following results: - The study’s PD program was implemented as intended. The PD providers delivered an average of 67.6 hours of PD per site, compared to 68 hours intended, and the treatment group teachers attended an average of 83 percent of the PD that was delivered. In surveys given to treatment and control group teachers, treatment group teachers reported participating in 55.4 hours more mathematics-related PD than the control group teachers.
- The PD program did not produce a statistically significant impact on teacher knowledge of rational numbers (effect size = 0.19, p-value = 0.15). On average, 54.7 percent of teachers in the treatment group answered test items of average difficulty correctly, compared with 50.1 percent for teachers in the control group.
- The PD program had a statistically significant impact on the frequency with which teachers engaged in activities that elicited student thinking, one of the three measures of instructional practice used in the study (effect size = 0.48). This measure encompasses such behaviors as asking other students whether they agree or disagree with a particular student’s response and also includes behaviors elicited from the students such as offering additional justifications or strategies. Treatment teachers on average engaged in 1.03 more activities per hour that elicited student thinking. The PD program did not produce a statistically significant impact on the other two measures of instructional practice: Teacher uses representations (effect size = 0.30; p-value = 0.0539) and Teacher focuses on mathematical reasoning (effect size = 0.19; p-value = 0.32).
- The PD program did not produce a statistically significant impact on student achievement (effect size = 0.04, p-value = 0.37).
|
You can read the full reports at: http://ies.ed.gov/ncee/pubs/20084030/index.asp and www.mdrc.org/publications/552/overview.html.
16 comments:
Both of these studies provide disappointing results, certainly. Are they applicable to Amherst, though?
Doesn't much matter since the teachers union gave up their 3 paid professional development days next year to save $350,000 (although they will take them as days off of course) in order to pass the Override.
This is fantastic. Professional development means nothing to student learning.
Would you mind doing some investigative reporting and telling us what does mean something to student learning?
Is every study you find valuable or only the ones that support your views?
What are you going to tell us next, that school readiness is not all it's cracked up to be?
Mr. Sam Clemons,
From your comment it seams as though you are implying CS is stacking the deck. I disagree.
I believe CS has intellectual integrity, a keen mind and ample experience to evaluate the research, and conviction to follow the evidence and formulate policy supported by facts and other values.
You have the means to contribute to the debate. Read the primary material, evaluate the studies for yourself. Determine if the studies are well constructed, draw appropriate conclusions and are absent of flaws that skew outcomes.
Then you can argue on the merit as opposed to ad hominem.
Best,
TomG
Sam Clemons or Sam Clemens?
I am probably going out into harms way on this, but I will say that during my entire K-12 career, there was only one professional development day that was worthwhile, and that was because my sponsoring teacher (this was during my student teaching) was a retired Army Bird Colonel and said "Ed, that is a good question, why don't you stand up and ask it?
And this was 20 years ago, but I stood up and asked "what happpens when the AIDS virus starts mutating, as viruses are known to do?" Things got interesting, people started learning things. And the so-called "experts" from Augusta had to actually earn their money....
We all know that educational quality has a lot to do with the quality of your teachers. Should anyone be surprised to learn that the quality of your inservice workshops also depends on the quality of those providing it?
And thus the question I ask is were these inservice events ineffective because such things are inherently ineffetive, or was it because they were poorly done?
There is a real question here: who is accrediting in-service workshops? Who is deciding that they are even worth attending?
I believe that longer school hours and a longer school year have a positive impact on student learning which is why President Obama is promoting both. I was disappointed to see that the Amherst School Committee did not raise this issue, especially with respect to early release Wednesdays. Not only does that give our kids less instructional time but also operates contrary to our social justice mission as it places a financial hardship on families who must find after school care for their children on Wednesdays.
A couple of things i'll divide into 2 posts:
First, FR parent--This would be a good opportunity to clarify some misconceptions about Wednesday afternoon. About 25 years ago (I don't know the exact year), there was a secondary school related bus issue (change of routes? I'm not sure) that necessitated a 20 minute delay at the end of the elementary school day. Of course, that needed negotiation, and the result was a "time payment" rather than a $$ increase for extra time. The 100 (5x20) minutes were subtracted from Wednesday afternoon--that is, the school week is the same number of hours as before the schedule change. Moreover, that time has become sancrosanct for elementary teachers. It is basically the only sustained planning and collaboration time available to them during the week, and the only predictable planning time. Many of the teachers' contracted daily planning time blocks (when their class goes to Specials) get sucked up by other things. So to summarize, it is not "less time" for schooling, there is a reason it is this way, and that time has become an extremely valuable time for our schools and teachers.
I may have accidentally posted my first part as Anonymous a minute ago.
Regarding professional development: Frankly, my experience in Amherst was that PD was a pretty big waste of time. It often did not address the issues that were most pressing to teachers and there was often little input sought from staff.
Any PD that is a one-shot deal without on-going support is pretty useless anyway, as CS's posted studies show (and amost any teacher will tell you). Coaching, however, does have an effect; of course, the quality of what teachers are trained in and the quality/focus of the coaching matters a lot to the results, and that may be less described in those studies. I know for a fact that good coaching has been shown to effect classroom change. If teachers are coached to teach a particular program, it may well not have much impact, but if the coaching is to understand students, the teaching/learning process, and the subject matter better, it usually does. Collaboration between teachers also matters. I don't have the time or energy right now to hunt those studies I've read in the past down, but I will look for them next week if I can.
I actually think it would be kind of silly to take an ideological stand that NO kind of help to teachers to help them improve their practice matters, or that all PD is just a "feel good" thing to do. In short, on-going, quality PD through collaboration and coaching is a positive thing, but most districts don't want to pay for it.
To Sam Clemons: if you can find serious academic research that shows that professional development increases student achievement, please share it.
Teacher (and staff) quality DOES matter. So does giving the teachers and staff enough freedom to innovate (as long as they're rewarded for success and punished for failure).
Amherst is lucky to be a very pleasant place to live, so we get pretty good teachers. Still, I think our school system would be much better if we ripped up the current union contracts and implemented policies that actually encouraged the bad teachers to find another job and rewarded the good teachers.
I'd personally start by eliminating raises for completing advanced degrees, because there's basically no correlation between having a Masters degree and being a better teacher.
PS: there's an article in this week's NY Times magazine that is relevant to this discussion (The Teacher Union's Last Stand: http://www.nytimes.com/2010/05/23/magazine/23Race-t.html?ref=magazine )
I agree with the thought that the best thing for all our kids would be an extended school day and/or extended school year. Not only would it help achievement but it would also help support our social justice mission by reducing the amount of after school and summer care students need (harder for lower income families) and by reducing the summer knowledge loss that disproportionally affects those students who are struggling in the first place.
Massachusetts (along with California) has been among the leaders in the new Expanded Learning Time Initiative with good results. A comprehensive article on the subject can be found at: http://www.americanprogress.org/issues/2010/02/transforming_schools.html
As the article points out, though, it is not just additional TIME that has helped but also a redesigned curriculum (which includes an expansion of subjects taught). I would love to see our School Committee(s) take up this idea. It is also another good reason for a K-12 district since it would be much easier to implement these changes if there were curricular alignment all the way from K-12 and all four towns were supportive of the transformation.
Do college professors benefit from professional development? attending conferences?
My response:
Anonymous 8:23 - well, I think this research is compelling, and I'm not sure why it wouldn't apply to Amherst? Both of these studies were done in moderately or high poverty schools, where my intuition would say you'd get an even stronger effect of professional development than in low poverty schools. These are both well done studies which include appropriate controls and analyses. It is hard for me not to take these results seriously.
Larry - good point (although that was just for one year). But it certainly suggests to me that teachers aren't seeing those days as very valuable, if they are willing to give them up!
Sam Clemons - I really debated about whether to even post your comment, since it is so non-constructive. But I did, largely to illustrate the type of reaction I get from simply posting a research study. Remember, I didn't run these studies, I wasn't involved in these studies -- I'm just sharing research on education issues, which I would thing would be of interest to readers of my blog. I'm sorry this research goes against your intuition ... does that mean it must be wrong? Feel free to send links to published studies describing the benefits of professional development ... I haven't found those, and these papers both got a lot of attention in Education Weekly.
Tom G - thanks for the support. I share your disappointment in Sam Clemons' approach to discussion.
Ed - good questions, and it is just hard to tease out whether it was the content or the educator.
FR Parent - the length of school day (including Wednesday afternoons) is a contract issue with the teachers. There is no way we could raise this now, since the contract is in its third year. If you believe this is something you'd like to see changed, please send an email to the whole SC (schoolcommittee@arps.org) and I can then ask for this to be on the agenda for discussion (remember, this would involve lengthening the school day for teachers, so it would involve more $$).
Ken - thanks for adding the historical perspective on the Wednesday afternoons. And it seems like your experience is in line with that shown in this study -- professional development doesn't seem to have a large effect. However, these studies DID consider coaching, and they didn't find effects of coaching. If you know of other research showing coaching matters, please send me a link.
However, I'm not taking "an ideological stand" about professional development -- I'm saying SC members and administrators and principals should be familiar with the most current research on education issues to make sure that schools are using limited resources wisely. These two studies (one involving math, one involving reading, examining a number of schools in 18 different districts) seriously call into question whether professional development is effective, and I think that is very important for us to take seriously.
More from me:
Gavin - thanks for your post. And your point about why we pay extra for advanced degrees is well-taken: this is precisely an example of something that "seems" like it would matter (e.g., teachers with masters must be better than teachers with bachelors), yet doesn't.
Alison - the issue of extended day is actually something that Amherst has discussed (it was mentioned several years ago as part of the reorganization committee I was on, prior to SC). However, it is expensive (e.g., you have to pay teachers more), and thus the state has been supporting it with grants for districts that are at risk (e.g., high poverty). Amherst isn't going to be eligible for that money, so I think we'd need to think carefully about what we gave up in order to have more school time (e.g., are we willing to have larger class sizes)? I agree that this could have benefits, especially for kids who don't now have good afterschool activities. Unfortunately, I can't see K to 12 regionalization with the small towns in our area occurring in my lifetime (though I agree that this would be beneficial) ... I think there is very little interest from the other towns, and the rules for accomplishing this are VERY complicated!
Anonymous 7:45 - absolutely: I really depend on attending conferences for my own "professional development" and that is why I would NEVER support a request from the Amherst College administration to give up this support ($1,000 per year). Last year no faculty at Amherst College got raises -- yet we all continued to receive our professional development funds -- and there were no faculty complaints. Yet the Amherst teachers voted to give up their professional development days and maintain their raises (3%) -- which to me indicates that the teachers didn't see missing the professional development days as essential for their own development. But this is also pretty different, because our professional development funds are NOT about improving our teaching; they are about working on research activities (e.g., presenting our work at conferences, collaborating with colleagues at other institutions, etc.), so this isn't really comparable. I'd be surprised if professional development had a major impact on teaching in the college classroom either.
Catherine,
I wasn't accusing you personally of taking an ideological stand. I was speaking in general, whether you, other posters, or anyone else might conclude that, by definition, no PD has value to teachers based on those studies. In any PD study 4 things are being measured: 1) the professional development model, 2) the efficacy of the particular areas teachers are getting PD in, 3) the efficacy of those particular trainers and coaches, and 4) teacher buy-in. I would have to look at the studies more carefully to see how they differentiated these 3 things.
In my own case, I was trained in a cognitive education program, and after 2 years of applying that training with a small group of 3 struggling learners, their LA and Math MCAS scores jumped from 220-225 in 4th grade to 240+/- in 6th, and in a 4th grade classroom (22 students) I used it in, every student's MCAS math score improved from the previous year. So I know that the right kind of training in the right kind of thing can have very powerful results.
I understand that recently there has been some dirt kicked up at the Mark Twain gravesite at Woodlawn Cemetery in Elmira, New York, occurring just as someone posting as "Sam Clemons" has begun expectorating regularly on this blog.
Mr. Twain wants us all to know that the "o" where a second "e" would be in the last name should eliminate any confusion in the poster's identity.
Furthermore, relevant to "Sam Clemons" specifically and life in general, Mr. Twain commented that "when we remember that we are all mad, the mysteries disappear and life stands explained."
This is the best evidence that I am aware of that Mark Twain may have visited Amherst, Massachusetts and sampled our political life during his time above ground.
Post a Comment