Showing posts with label data-driven decision making. Show all posts
Showing posts with label data-driven decision making. Show all posts

Wednesday, June 27, 2012

Everything is Subjective




This morning I read a post by Russ Goerend that hits on a lot of issues that have been going through my mind lately and the subsequent conversations on Twitter around the topic have clarified my thinking a little bit more. The topic is data.

In Russ' post he embeds a Tweet from another educator whose logic bothers him:



To which Russ replies:
If that’s what you are without data, what are you with data? “Just” a person with an opinion and data? Just a person with data?

This led to an even more interesting thought when Russ' post was discussed on Twitter:



William didn't reply so I felt the need to butt in. This line of reasoning is what I have been thinking about lately and felt this the logical place to express this idea:





To which set off a debate on the semantics of which was opinion, the data itself or the act of collecting it which to me is not nearly as important as the fact that this all calls into question the nature of data and the value of objectivity. Continuing to think through this conversation today I came to this thought which had not occurred to me before:






Two books I have recently read (actually one I am half-way through right now) address this issue but never quite nail it down. First, in Gerald W. Bracey's (2009) Education Hell: Rhetoric vs. Reality: Transforming the Fire Consuming America's Schools Bracey discusses how data can be twisted around to make make arguments that the data itself doesn't necessarily support. One example he gives is the data that lies at the heart of the (1983) commission report, A Nation at Risk, which arguably set off much of the data-driven madness we are currently under. In that report Bracey says there was a data set consisting of 9 points (3 tests given to 3 different age groups). Of those 9 data sets all but one showed growth and improvement by students attending our school but the report only uses that one data set from that one test given to that one age group to base its claims that America's schools were in serious trouble. Here we have an opinion that is not only choosing the method of gathering data it is determining the data set upon which to focus attention. And, as so many education writers (Alfie Kohn, Diane Ravich, Herb Kohl, and Johnathan Kozol just to name a few) have pointed out, the instruments we use to collect this testing data are also flawed and reflect a cultural bias of the test makers. The data itself is boiling with bias and opinions.

Then, in Neil Postman's (1985) Amuzing Ourselves to Death: Public Discourse in the Age of Show Business, Postman points out that the nature of what we consider reliable data sources has changed over time. Once it was that “feeling is believing” then “saying is believing” then “seeing is believing” then “reading is believing” then “deducing is believing” and now “counting is believing.” Postman argues that it is the media driven culture that has reduced our concept of what is believable data to that which can be counted, that which can be objectified and abstracted.

So, as Postman points out, the nature of truth is flexible and as Bracey points out, there are ways to manipulate the truth. Among all this non-linear discourse on the matter was a conversation I had with Jennifer Borgioli stemming from and included in the conversation with William and Russ on Twitter. Jennifer nails the real issue with this tweet:



If you want to effect change the real issue is not how we use data. Data can be used as either a tool or a weapon but it is not the greatest source of power. Instead, we ought to be thinking about how to create new social constructs. Objectivity only exists within a context and objective measurement can only exist within that kind of frame of reference. Ultimately everything is subjective. Our collective fetish with data has caused many people great harm because it ends up being used as a weapon. Data is by nature an abstraction and when it is used to measure people it objectifies people; when people are objectified it opens them up to be exploited. The way to fight "counting is believing" is to construct a new context where something else is the arbiter of truth. What that looks like and how we get there I am not completely sure.






Reading is Believing:



Feeling is Believing:



Counting is Believing:



Never Tell Me the Odds!


Deducing is Believing:




Seeing is Believing:



That's no moon. It's a space station


Saying is Believing:

Posted by Carl Anderson at 2:49 PM 1 comments
Labels: , , , ,

Sunday, January 23, 2011

Making the wrong "Data-Driven Decisions"


Almost six years ago I heard the question asked for the first time in a job interview, "How do you use data to inform your decisions in the classroom?" I was a little taken back by the question and didn't really know what to make of it. I was interviewing for an art teacher position and considering the nature of the job and considering the questions I was accustomed to hearing in interviews for similar jobs, the question didn't seem like it was developed with the intent of finding the qualities they normally would look for in an art teacher. In fact, the question seemed a bit absurd.

I remember answering the question something like this:
"It depends on what you mean by data. Data could mean many things and can be acquired in many ways. If I have a field drawing lesson planned where I would take the kids outside to draw and it is raining, that data would tell me I probably ought to keep them indoors. Likewise, if I give an assignment for students to draw something in linear perspective and a kid draws nothing using the technique I use that data as an indicator that they probably didn't understand how linear perspective works and that further guidance would be necessary. If you are wondering how test scores affect how I make decisions in the classroom I don't have much to tell you other than the art teacher typically isn't one who is pulled into focus groups to study and analyze that kind of data."
I ended up getting the job. Turns out this question was one of those standard questions written by the district administration for use with all their teacher candidates for all positions. Also turns out that the school in this district that was hiring for this position was filled with teachers with an equal disdain and skepticism regarding what has come to be called "data-driven decision making."

Most administrators and bureaucrats love to hear that you are using data. I am thoroughly convinced that it doesn't matter what your conclusion is or how narrow your focus is, if you can use any set of statistical data to defend any practice it is good in their eyes. After all, if you have numbers you can prove something right? Problem is, in the classroom no amount of data can give a full picture. There are just too many factors and so many of them do not translate well into statistics. Decision making in the classroom requires the ability to manage a much broader and more complicate set of data than that of the statistician.

Data can often lead to poor decisions. Tonight I left the house briefly to make an emergency run to the store to buy a kit to fix my eyeglasses. The thermometer said -8 degrees Fahrenheit. At the store there was a guy in line in front of me wearing shorts. I asked him if he considered how cold it was outside before making his wardrobe choice. He said, "I put on what was clean." Obviously this guy made a data-informed decision. He examined his clothes and determined that shorts met the criteria of being clean so obviously they would be a good choice. He could defend his choice with data regardless of other data sets that would otherwise make someone else choose differently. I probably would have made a data-driven decision to put on a dirty pair of pants to run to the store if my choice was between clean shorts and dirty pants given the temperature outside. If this guy were an administrator and in charge of assessing other people's data-driven wardrobe decisions, would he give me a poor evaluation? Probably.

The same scenario is played out in education everywhere the misleading phrase "data-driven" rears its ugly head. Its not whether you use data to inform your decisions or not, everyone does this. Unless you are making a blind choice or using a coin to determine the outcome of a decision you always use some kind of data to inform that choice. Heck, even the outcome of a coin toss is a form of data. What matters more is what qualifies as data and in an organization today that emphasizes this phrase usually that data set is intentionally narrow. Narrow sets of data lead to foolish decisions, decisions that are still data-driven. In practical terms, this phrase is meaningless.

So, what are some bad decisions you have seen schools make that were "data-driven?"



This post is fifth in my "war on words" series. Other terms in this series are: "best practices," "child-centered," "value added," and accountability.
Posted by Carl Anderson at 12:43 AM 1 comments
Labels: , , ,
Subscribe to: Posts (Atom)