Why We Need Data

The single most frustrating thing to me (and you might think this would be a hard call) about my experience with the Amherst schools is what I see as a total reluctance, and indeed almost opposition, to looking at DATA. If you even mention wanting data, you get accused of all sorts of evil things -- being elitist, having a PhD, working at a wealthy institution, living in Amherst Woods, caring only about numbers, and teacher-bashing. But in all seriousness, we can see many, many examples of times in which relying on anecdote and intuition is really problematic.


One of my favorite examples comes from of the hallmarks of the Bush presidency -- a focus on abstinence-based sex education as the best approach to preventing teenage pregnancy (here is a case in which most people agree that unwanted teenage pregnancy is a good thing to prevent, but people disagree with how we go about this prevention). Under Bush's leadership, the federal government provided states with considerable sums in order to teach youth about abstinence from sexual activity outside of marriage (many states ended up turning down the funds because they prohibited comprehensive sex education). But when research examined the effectiveness of abstinence-only programs, it revealed that participants had just as many sexual partners as nonparticipants and had sex at the same median age as nonparticipants (http://www.mathematica-mpr.com/abstinencereport.asp). As stated by Dr. Ned Calonge, Colorado's chief medical officer, "To show no benefit compared to nothing. That was striking. These are tax dollars that are going for no useful purpose, and it would not be responsible for us to take those dollars."


Here's another example from a book called Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-Based Management (by Jeffrey Pfeffer and Robert I. Sutton). (And for those who are watching for my elitism, let me be upfront -- these authors teach at Stanford University, where I went to college).


"Almost any decision you make about any sort of intervention can be evidence-based in the sense that you can try to access what research literature and evaluation literature demonstrate about this. It’s a way of thinking more scientifically and systematically. An individual doctor can’t tell if Vioxx causes heart problems because one person’s practice isn’t big enough to determine that evidence. It’s similar if you’re one manager thinking about making a merger decision, or implementing ERP [enterprise resource planning], or putting in incentive-based pay; a good manager will look at larger evidence in terms of informing the decision.


The way a good doctor or a good manager works—we call it the attitude of wisdom—is to act with knowledge while doubting what you know. So if a patient goes to a doctor, you hope the doctor would do two things: first look at the literature and make the best decision given what’s available. Then actually track the progress of the treatment and see what unexpected side effects you’re having and what things are working."


OK, so I think we can pretty easily imagine the parallels to what we do in the Amherst schools here. First, asking one intervention teacher how effective he/she is at improving outcomes is just not a large enough sample to provide evidence of such effectiveness, nor is spending a day at the middle school observing classes, nor is visiting a small school for a day to see the beneficial outcomes. You get evidence by gathering data from multiple sources as reported in the literature (and there is a lot of literature on issues in education, although some of it is flawed in various ways), not through personal anecdote and intuition on the part of a very small number of people in one school system, and then you make the best decision you can based on that evidence. Second, you actually track the progress of the treatment and see what things are working and what things aren't (e.g., how is the new 9th grade science program working; what is the impact of having a trimester versus a semester system; how well is the "extensions" model of differentiating math instruction in 7th grade working; how well do our programs for underperforming kids help them ). Then, you either drop things that aren't working (and you can have clear and public criteria for what "working" means) or you modify them to see if they do work (and again, you are clear and transparent about what is and is not working and what will be done to improve outcomes).


OK, the key question for me is how do you get organizations (or, in this case, a school district) to start using evidence-based decision-making? This is what Pfeffer and Sutton say: "Encourage people to be noisy and nosy—it promotes wisdom." Here's a real world example described in their book: "Researchers in one study examined 194 patient care failures by nurses, everything from problems caused by broken equipment to drug treatment errors. This work revealed that those nurses whom doctors and administrators saw as most talented unwittingly caused the same mistakes to happen over and over. These “ideal” nurses quietly adjust to inadequate materials without complaint, silently correct others’ mistakes without confronting error-makers, create the impression that they never fail, and find ways to quietly do the job without questioning flawed practices. These nurses get sterling evaluations, but their silence and ability to disguise and work around problems undermine organizational learning."


This finding, and hundreds of others from research studies, led to the development of a model they describe as "The talents of wisdom: People who sustain organizational learning". Here are the types of people they believe are most beneficial to such a process:

• Noisy complainers - Repair problems right away and then let every relevant person know that the system failed
• Noisy troublemakers - Always point out others' mistakes, but do so to help them and the system learn, not to point fingers
• Mindful error-makers - Tell managers and peers about their own mistakes, so that others can avoid making them too. When others spot their errors, they communicate that learning—not making the best impression—is their goal
• Disruptive questioners - Won't leave well enough alone. They constantly ask why things are done the way they are done. Is there a better way of doing things?

"All of these characteristics help people act on what they know, and keep improving their own skills, peers’ skills, and organizational practices and procedures. The crux is, if you want better performance instead of the illusion of it, you and your people must tell everyone about problems you’ve fixed, point out others’ errors so all can learn, admit your own errors, and never stop questioning what is done and how to do it better. These actions can annoy doctors and administrators—or any other authority figure—who prefer quiet and compliant underlings, but if we want organizations that do as much good and as little harm as possible, these talents are essential."


I think this book (which I strongly recommend to those interested in thinking more about evidence-based decision-making in general) makes two key points that are relevant to the current state of our schools. First, it is really important to examine what people are doing and the outcomes of their behavior (meaning, it is not "nurse-bashing" to examine causes of patient care errors). Second, disruptive questioners (and I think I fit the bill pretty well here) are actually beneficial in terms of leading to better performance, instead of just the illusion of it (and I bet a lot of these disruptive questioners are seen as bullies). For a long time, people who raised any questions or concerns about the Amherst schools were seen as disruptive, divisive, only caring about their kids, elitist, and so on. But the reality is, our schools, like other organizations, really benefit from this type of an open dialogue, in which we are carefully considering what we are doing, and communicating this careful consideration broadly, not simply assuring everyone that the schools are great and that their kids will be OK.