Badges! Two: Why Data Is Never Enough

Yesterday, I posted on the ‘digital badges’ that some think may point the way toward an alternative form of educational certification. I don’t think that’s a good idea, but I didn’t really explain why (saying I would later get to why that’s not the same as a college degree). This morning, I saw a link on Diane Ravitch’s blog to an article by Esther Quintero, a sociologist who writes for the Albert Shanker Institute’s Shanker Blog. Entitled “The Data-Driven Education Movement,” it explains why data alone is insufficient for educational analysis.

Quintero points out what should be obvious but is not, that we rely to readily on data that can be quantified easily thus limited the data universe we consider, and that we assume incorrectly that there’s truth in data in and of itself. Thing is, as Quintero says:

(1) numbers are not the only type of data that matter; and (2) all data need to be interpreted before they can be elevated to the status of evidence – and theory should drive this process, not data.

I would add something else that Quintero talks about, (3) the fact that theory should drive the process is also a limitation–so theory needs to be constantly questioned as data is accumulated and assessed.

She uses as an illustration of the problem of our use of a limited data universe a story I know somewhat differently–but the point is the same. Someone who has lost something is seeking it under a streetlamp. When asked why there only, the person says, scornfully, that it is night, so they couldn’t see anything elsewhere… so why look where you can’t see?

Quintero then talks about how we often use data to develop theory rather than using it to test theory. The problem with this is that theory developed from a limited data set (and all are limited) is always going to be incomplete–even if it is consistent. Data, in other words, is better used for proving theory wrong than in generating theory out of whole cloth. Yes, it can be used as an impetus for modifying or rejecting theory, but no set of data alone should be the force behind creating theory. It’s just too limiting. Kurt Godel, in his famous proof, showed (to put it most simplistically) that no system can be both complete and consistent. If it is complete, it contains contradiction. If it is consistent, it does not cover all possibilities. What we have with theory that is solely data driven is movement toward consistency–but that, again, only accounts for what we can see under the streetlamp.

Quintero ends with this:

Our blind faith in numbers has ultimately caused impoverishment in how (and what) information is used to help address real world problems. We now apparently believe that numbers are not just necessary, but sufficient, for making research-based decisions.

The irony, of course, is that this notion is actually contrary to the scientific process. Being data-driven is only useful if you have a strong theory by which to navigate; anything else can leave you heading blindly toward a cliff.

Real education is a great deal more than reaching numerical benchmarks (though they do have their place). ‘Critical thinking,’ though a tired old phrase bowed to and ignored (for the most part) today–and not simply skills–is at its core. As ‘critical thinking’ deals with those things outside of the glare of the streetlamp, our data analyses can’t encompass it, so it gets passed by in favor of things that can be counted, things we now bow to as information that can be assessed repeatedly with the same conclusion reached each time.

That’s inherently regressive, always looking back, wanting to repeat what has been done before. The rubrics for writing exams, no matter how flexible we want to make them, always seem to drag the writers back to the five-paragraph theme. Why? Because they always focus on things that can easily be counted (paragraphs, sentences, spelling errors, sentence types, punctuation variety, supporting points, etc.). They have to, or the conclusions reached will be different for different readers.

The problem with that? Someone who writes brilliantly, but in another fashion, will always fail–even when the reader knows the writing is better than anything encompassed in the grading rubric. The reader isn’t given the freedom of deciding on his or her own, but only to make a decision based on the ‘countables’ of the rubric.

It is this freedom, one that allows for a real interaction between teacher and student, that is at the core of real American education–and is why our education is so good. And it is this freedom that digital badges (which only deal with what can be found under the streetlamp) do not provide.

And that, of course, is why digital badges will never be a real challenge to traditional structures of education–as long as those overseeing those structures don’t cave in to the contemporary mania for simplistic data-driven assessment.

7 thoughts on “Badges! Two: Why Data Is Never Enough

  1. Pingback: Barlow bashing data « Gas station without pumps

  2. Excellent. I was being irritated by the idea that every aspect of administration, curriculum, etc., should be “data-driven decisions.” Surely it has protected me (classes can be shown to suffer with large class sizes, composition is labor intensive, so clocking time can be quantified), but these make administration easier, not necessarily teaching and learning. The whole “best practices” idea is not what my best practices are, and what your best practices are, but what can be imposed by fiat because of a data set that didn’t include me.

    Neither am I asked if it fits my classes. To be fair, I’m given various options that I can use, and in the scheme of things nothing is too onerous yet, but administration intervention, usually to support metrics for accreditation or some comparison so our school can be seen as more deserving of monies than another, are well-intentioned interventions nevertheless, and can be disruptive.

  3. Pingback: Badges! Two: Why Data Is Never Enough | Academe Blog | Badge Talk |

  4. Pingback: Badges! Two: Why Data Is Never Enough | Academe Blog | The Daily Badger |

  5. Pingback: Badges! Two: Why Data Is Never Enough | Academe Blog | Badges for Lifelong Learning |

  6. Of the thirty projects funded under the Badges for Lifelong Learning initiative, not a single one is aiming to use badges in the manner you describe. If you could simply “swap” badges for grades then they would not make a difference. But deciding what to recognize and how to assess it in the forward-facing context of open badges is far more than giving grades. Whether or not THAT transforms formal education wholesale remains to be seen, but it has certainly already started to transform informal education and some key aspects of formal education

Your comments are welcome. They must be relevant to the topic at hand and must not contain advertisements, degrade others, or violate laws or considerations of privacy. We encourage the use of your real name, but do not prohibit pseudonyms as long as you don't impersonate a real person.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s