My Goal in Blogging

I started this blog in May of 2008, shortly after my election to the School Committee, because I believed it was very important to both provide the community with an opportunity to share their thoughts with me about our schools and to provide me with an opportunity for me to ask questions and share my thoughts and reasoning. I have found the conversation generated on my blog to be extremely helpful to me in learning community views on many issues. I appreciate the many people who have taken the time to share their views. I believe it is critical to the quality of our public schools to have a public discussion of our community priorities, concerns and aspirations.

Monday, May 24, 2010

Second-Class Science: Education Research Gets an F

by Sharon Begley
Published Apr 29, 2010
From the magazine issue dated May 10, 2010

Since holding teachers responsible for student performance is now all the rage, from the White House to the political right, let us do a simple thought experiment. Imagine an amateur baseball league in which team owners dictate which bats players use. The owners try to choose the best, but the research on bats is so poor, they have to rely on anecdotes—"Barry Bonds hit 73 home runs with maple!"—and on manufacturers' claims. As a result, some teams wind up using bats that are too heavy, too fragile, or no better than a broomstick. Does it make sense to cut players who were forced to use ineffective equipment?

It goes without saying that effective teaching has many components, from dedication to handling a classroom and understanding how individual students learn. But a major ingredient is the curriculum the school requires them to use. Yet in one of those you've-got-to-be-kidding situations, the scientific basis for specific curricular materials, and even for general approaches such as how science should be taught, is so flimsy as to be a national scandal. As pay-for-performance spreads, we will therefore be punishing teachers for, in some cases, using the pedagogic equivalent of foam bats. "There is a dearth of carefully crafted, quantitative studies on what works," says William Cobern of Western Michigan University. "It's a crazy situation."

Cobern tried to fix that in a study comparing direct instruction with inquiry learning, competing ways to teach science. The smart money has been on the latter, in which students explore a question on their own by, say, growing some seedlings in a closet and others on a windowsill to discover photosynthesis rather than being given the concept by the teacher. Cobern's team randomly assigned 180 eighth graders (randomization is the gold standard for research, as in trials for new drugs) to one or the other form of instruction, they report in a study published in Research in Science & Technological Education in April. Contrary to received wisdom, "as long as students are actively engaged, direct instruction does just as well as inquiry-based teaching" in how well kids learn science concepts, he told me. Yet national and state standards push inquiry learning. As Cobern's team diplomatically put it, "Some claims for inquiry methods regarding understanding the nature of science are not sufficiently supported by evidence."

Indeed, an exhaustive analysis of 138 studies of inquiry-based science instruction in K–12 found that most of them had highly problematic designs: 53 percent did not randomly assign students to one kind of instruction or failed to include a control group. Not only did most studies have "marginal methodological rigor," the analysts found, but the trend was "toward a decrease" in rigor.

When it comes to specific curricula, the scientific vacuity of education research is even more exasperating. In 2002 Congress established the Institute of Education Sciences in the Department of Education to conduct peer-reviewed studies of everything from math software to rewarding kids for academic performance. If you have a child in K–12, peruse IES's What Works Clearinghouse. It is an ugly picture. Only one of five studies of Saxon Math (a home-schooling program for grades 6 to 8) met the standards for scientific rigor, IES found. For Web-based Odyssey Math, a K–8 curriculum, no studies out of 20 did so. Of 12 studies of Singapore Math, modeled on that city's supposedly superior math program, not a single one was methodologically rigorous (in many cases, there was no comparison group). Of 23 studies of Bridges in Mathematics, not a single one met evidence standards, and 16 were so sloppy that it was "impossible to attribute the observed effect" to the program. None of the 40 studies of Investigations in Number, Data, and Space met the standards.

And where the studies were rigorous, the curriculum often flunked. None of four LeapTrack math programs "demonstrated significant effects on student achievement." For Plato Achieve Now, which runs on a PlayStation Portable and emphasizes learning at one's own pace, there was "no discernible effect on math achievement." Ready, Set, Leap!, a preschool reading curriculum, "was found to have no discernible effects on oral language, print knowledge, phonological processing, early reading/writing skills, and math," based on two good studies. And so on.

Philosopher Alfred North Whitehead wrote that when one compares the importance of education with "the frivolous inertia with which it is treated," it is "difficult to restrain within oneself a savage rage." That was 80 years ago.

Sharon Begley is NEWSWEEK's science editor and author of Train Your Mind, Change Your Brain: How a New Science Reveals Our Extraordinary Potential to Transform Ourselves.


Anonymous said...

In an earlier post "Comparing School Unions" in response to your statement "so hopefully someone will run against me", a reader asked "Have you decided to run for re-election?"

Will you be running for re-election?

Catherine A. Sanderson said...

Anonymous 9:03 - I'm absolutely running for re-election (as I've announced earlier on this blog), and my term is up in the spring of 2011. I look forward to an active and energetic campaign, hopefully with many candidates, so that voters can choose which candidate they'd like to serve on the SC.

Anonymous said...

For reading methods, I'd like to see a study that compares reading to babies and toddlers while they sit in a lap against any other reading program in schools.

Abbie said...

I wonder how much 'real' research has been done on tracking...By real, I mean with (proper) controls and good experimental design.

Anonymous said...

I teach remedial ELA, and sometimes I have to use a computer program instead of doing my own teaching. I'm always happy to take the kid off the computer and do some of what I think of as "real teaching" where the student and I INTERACT and I can shift gears up and down constantly as I see how he or she is processing the material.

But "my way" is untested, supposedly the computer-based remedial program has "proof" that it works.

I give up.

Anonymous said...

I'm very curious about how you reconcile this article, which is very critical of the reliability of educational research, with your plea for the Amherst Schools to base more of their educational decisions on said research.

Anonymous said...

I don't remember CS asking that decisions regarding the Amherst schools be based on educational research. What she asks is that we make decisions based on evidence from what works in other schools similar to ours.

Anonymous said...

Wasn't an article about a study finding little if any positive gain from prof. development just recently posted? With comments from her like, "... I'm not taking "an ideological stand" about professional development -- I'm saying SC members and administrators and principals should be familiar with the most current research on education issues to make sure that schools are using limited resources wisely."

I'm just wondering how she reconciles that sentiment with the article about the general inadequacies of most educational research. How are we to know which studies are worthy of trusting?

Catherine A. Sanderson said...

My response:

Anonymous 12:32 - I bet this reading program would be far better than any provided in elementary school ... however, I don't think this is an approach the Amherst public schools has control over.

Abbie - great question! I am actually preparing a blog posting on a recent article on tracking that attempted to use more controls than prior research. It is clear that a lot of the research on tracking isn't well done.

Anonymous 6:05 - I'm not sure why you say "I give up." There is value in research because it is broader than just one single person's experience and their subjective determination of how well it worked. That is why we use research to figure out how to combat cancer -- we don't just say "well, I did this, and my patient seemed to like it, so I'm going to keep using it." Research allows us to determine whether something is actually effective -- and I would think all teachers would welcome this information.

Anonymous 6:59 - two things. First, I believe this article is acknowledging that most research on education is poorly done -- and hence we shouldn't rely on poorly done research! Second, I believe that some research (like that cited in this article) is in fact VERY well done, and we should certainly rely on that in terms of making decisions. It is not hard to figure out the difference between well done and poorly done research -- was there a control group, was there random assignment, etc. That is precisely the point this article makes.

Anonymous 2:08 - thanks for adding this. I believe we should rely on research AND the experience of other districts. I wish this district would choose to rely on both more than they currently do, although I don't see a lot of move in this direction.

Anonymous 7:01 - I believe that we should rely on research to decide how to allocate resources. Well done research has two basic features: there is a control group (e.g., one group gets something, one group gets something else, so you can compare them), and random assignment (e.g., schools/teachers/parents don't choose who gets what). When you don't have these two features, you can't really draw conclusions (although if you have a control group and don't have random assignment, at least you can test statistically to determine whether particular factors are responsible for any effects).

Anonymous said...

Anon 7:01 said: "How are we to know which studies are worthy of trusting?"

The answer to that question is that if a study proves a point or position CS is trying to make then its trustworthy. If a study disproves a point or position CS is trying to make then its untrustworthy.

It's quite simple, really.

Catherine A. Sanderson said...

My response:

Anonymous 7:25 - actually, I didn't make up the standards for determining whether a study is good ... others have, and it isn't a great mystery. As the article states, "randomization is the gold standard for research, as in trials for new drugs." It is fun to criticize me anonymously, I'm sure -- but the features of a well-designed research study are very evident, and could easily be identified not only by me, but MANY others (e.g., I'm sure this is covered in high school science classes, and potentially even middle school science classes).

Anonymous said...

To Anon May 26, 2010 7:25 AM

Then I guess you don't like that medical research is based on studies. You can't pick and choose who you want to do studies and who you don't want to do studies. The only way to move forward is to have studies, have them peer reviewed, and have meta analysis. Life is constantly evolving. There never is a definitive "this is the way it's done", period.

Harekrushna Behera said...

I am Harekrushna from India. I m very interested in educational research. I m a school teacher.
Very happy to see ur blog.
Guide me to do something in education....