Making the wrong "Data-Driven Decisions"

Almost six years ago I heard the question asked for the first time in a job interview, "How do you use data to inform your decisions in the classroom?" I was a little taken back by the question and didn't really know what to make of it. I was interviewing for an art teacher position and considering the nature of the job and considering the questions I was accustomed to hearing in interviews for similar jobs, the question didn't seem like it was developed with the intent of finding the qualities they normally would look for in an art teacher. In fact, the question seemed a bit absurd.

I remember answering the question something like this:

"It depends on what you mean by data. Data could mean many things and can be acquired in many ways. If I have a field drawing lesson planned where I would take the kids outside to draw and it is raining, that data would tell me I probably ought to keep them indoors. Likewise, if I give an assignment for students to draw something in linear perspective and a kid draws nothing using the technique I use that data as an indicator that they probably didn't understand how linear perspective works and that further guidance would be necessary. If you are wondering how test scores affect how I make decisions in the classroom I don't have much to tell you other than the art teacher typically isn't one who is pulled into focus groups to study and analyze that kind of data."

I ended up getting the job. Turns out this question was one of those standard questions written by the district administration for use with all their teacher candidates for all positions. Also turns out that the school in this district that was hiring for this position was filled with teachers with an equal disdain and skepticism regarding what has come to be called "data-driven decision making."

Most administrators and bureaucrats love to hear that you are using data. I am thoroughly convinced that it doesn't matter what your conclusion is or how narrow your focus is, if you can use any set of statistical data to defend any practice it is good in their eyes. After all, if you have numbers you can prove something right? Problem is, in the classroom no amount of data can give a full picture. There are just too many factors and so many of them do not translate well into statistics. Decision making in the classroom requires the ability to manage a much broader and more complicate set of data than that of the statistician.

Data can often lead to poor decisions. Tonight I left the house briefly to make an emergency run to the store to buy a kit to fix my eyeglasses. The thermometer said -8 degrees Fahrenheit. At the store there was a guy in line in front of me wearing shorts. I asked him if he considered how cold it was outside before making his wardrobe choice. He said, "I put on what was clean." Obviously this guy made a data-informed decision. He examined his clothes and determined that shorts met the criteria of being clean so obviously they would be a good choice. He could defend his choice with data regardless of other data sets that would otherwise make someone else choose differently. I probably would have made a data-driven decision to put on a dirty pair of pants to run to the store if my choice was between clean shorts and dirty pants given the temperature outside. If this guy were an administrator and in charge of assessing other people's data-driven wardrobe decisions, would he give me a poor evaluation? Probably.

The same scenario is played out in education everywhere the misleading phrase "data-driven" rears its ugly head. Its not whether you use data to inform your decisions or not, everyone does this. Unless you are making a blind choice or using a coin to determine the outcome of a decision you always use some kind of data to inform that choice. Heck, even the outcome of a coin toss is a form of data. What matters more is what qualifies as data and in an organization today that emphasizes this phrase usually that data set is intentionally narrow. Narrow sets of data lead to foolish decisions, decisions that are still data-driven. In practical terms, this phrase is meaningless.

So, what are some bad decisions you have seen schools make that were "data-driven?"



This post is fifth in my "war on words" series. Other terms in this series are: "best practices," "child-centered," "value added," and accountability.