Ask a stupid question, get a stupid educational system

I’ve spent the past few days reading Clifford Hill’s work on how children interpret questions on reading tests, and I can’t help but be struck by the comparisons to what Ethan is writing about his students in Cameroon.

Hill demonstrates again and again that children generate their answers not just from the text of the very brief, decontextualized passages which show up on the tests, but from a good deal of extra cognitive work they do in order to generate a mental system in which the passages make sense, or connect these passages to the world they know. Here is one of his examples of a test passage and questions, from the Lawrence Cremin Lecture he delivered at Teachers College this past March:

Raisins are made from sweet varieties of grapes. The ripe fruit is usually placed on trays right in the vineyard. There, it dries in the sun. Drying may take several weeks.

A. Raisins are made from grapes that have a lot of
water     varieties
skin        sugar

B. What kind of climate is best for making raisins?
warm and dry     warm and wet
cool and dry         cool and wet

(If you’re not yet reminded of how soul-crushingly insipid most multiple-choice tests are, Hill’s written whole books on why they make for bad assessments. And actually, reading chapters upon chapters of children’s justifications for wrong answers is pretty entertaining, and will bolster your faith in the ingenuity and resourcefulness of your fellow person.)

Of questions like these, he says,

tasks which force children to manipulate polarities often engender a confusion that they would not experience when reading an ordinary text in which polarities are not overworked and are embedded in a naturally occurring context.

He provides a few examples of students whose thinking about these questions led them to choose a distractor answer:

“Look, first I didn’t understand so much so I picked warm and dry be-cause of this [points to “dries” in the text]. But then when I read it more, I think—uhm—warm and wet, ’cause here it says, “There, it dries in the sun. Drying may take several weeks.” Weeks. I think weeks are very long so I don’t think dry is very good.”

“Just as the extended drying was used to justify the choice of warm and wet in Task (B), so it was used to justify the choice of water in Task (A). As one girl put it with great confidence, “It has to be water because of all that drying in the sun.” Another girl read out loud the final sentence in the passage to justify her choice of water. “It was water, because here it says, ‘Drying may take several weeks.’” “

(As Hill notes elsewhere, correct answers like “varieties” are kind of puzzling in and of themselves; the word is pretty much peripheral to the whole point of the passage. I think this is what he was calling “acommunicative,” as opposed to “communicative,” questions.)

Hill concludes:

These girls, like the boy described in the previous paragraph, adopt a dynamic stance toward textual information, which clearly runs counter to the static orientation that testing calls for. This stance leads them to project from one lexical pole to its opposite.

These examples clarified for me one recent post of Ethan’s. He reports the following exchange:

Teacher: OK, class. What color is the sky?

Student, raising hand and being called upon, says: Air!

Teacher: OK, good, the sky has air in it. But what color is it?

Different student, too excited to wait to be called upon: Monsieur! It’s air!

Ethan’s been frustrated by what he has called the “illogical” nature of answers like this. But in this answer, as in some of the others he’s blogged about, there is a certain logic at work. The students did not excitedly raise their hands to call out, “Monsieur! It’s octopus!” They gave an answer related to the topic he was asking questions about, in a way that recalls Hill’s students generating a context for their answers which just happens to not be the one called for by the test makers.

One possible imagining of the students’ imagined context in this case goes like this:

“I am in computer class with Msr. Ethan. Computers are a difficult subject. Msr. Ethan is from America. He is a teacher, and very smart, because he understands things like computers and science.

“Msr. Ethan has asked us a question about the sky. Teachers like him often ask difficult questions about science. I need to give the most erudite answer I can to impress him, because heaven knows a teacher wouldn’t ask something completely stupid and obvious to all of us, like what color the sky was. Who even cares what color the sky is? The context of this question is school, not the world out in the fields.”

This is, of course, not taking into account any possible linguistic confusion. I have no idea if the local dialect’s word for “blue” happens to be the same word for “air.”

(Ethan was, of course, really focusing more in that post about the theme of re-trying strategies that have already proven not to work; but then, Cameroonians don’t have a lock on that market. And Ethan would probably say, as usual, that I don’t understand what’s going on there; but then, by his own admission, he doesn’t either.)

Does this give any guidance to Ethan so that he might elicit the answers he wants next time? Maybe not, except that it might be a good time to set a rule like “before you answer this a question, I want you to listen closely to exactly what I am saying, think, and then wait five seconds before you raise your hand.”

But Hill’s work goes a long way towards demonstrating the logic of wrong answers. My particular interest has been the way he illuminates fundamental linguistic differences in cultures’ ways of understanding simple spatial concepts like “in front of,” “behind,” “at,” “in,” and “on.” While he explains these primarily in terms of the ways it leads African and Chinese students to answer test questions differently, I think it can also tell us a good deal about how people understand where they are on the Internet.

Comments 4

  1. Roger wrote:

    (reposting from previous comment elsewhere) Sometimes I think the entire enterprise of schooling-as-exercising-authority is about avoiding genuine questions. There’s some great stuff in my old favorite teaching book, _Teaching as a Subversive Activity_, about this — how hard it is to ask students genuine questions, and how easy it is to substitute fake, stupid ones that cut off inquiry rather than inviting it. That’s so clearly the problem in your example: the students think the question is more interesting, harder, more genuinely meant, than the banal, unimaginative tester who wrote it.

    Also, if you haven’t checked out Banesh Hoffman’s delightful old screed _The Tyranny of Testing_, do.

    Posted 17 Dec 2010 at 1:30 pm
  2. gus wrote:

    I’m finding that I’m enjoying Hill’s book the way I enjoy my own dataset. I often end up laughing out loud. There’s something really wonderful about watching people make mistakes as they try to make sense; both his student test-takers and my blog readers are making beautiful hash. There’s something similar in it to enjoying Engrish.

    Posted 18 Dec 2010 at 1:19 pm
  3. Roger wrote:

    But those students aren’t failing! They’re making more sense of the question than the tester did!

    The passage does not actually provide *any* possible single correct answer to question B, since it gives no data on comparative raisin drying times in different climates. The students who guess “wet,” for instance, are making an absolutely legitimate guess that raisins would dry in days or hours, rather than the weeks described, in a dry climate. There’s no necessary reason that the “best” climate for raisin-making is the one where they dry fastest, either; the students may be assuming that the passage describes optimal drying times as slow. And as the analysis points out, question A clearly has more than one correct answer (at least “water” and “sugar” have to be right; the passage doesn’t provide any information for or against the other two).

    Posted 18 Dec 2010 at 2:41 pm
  4. Tomi wrote:

    “Msr. Ethan has asked us a question about the sky. Teachers like him often ask difficult questions about science. I need to give the most erudite answer I can to impress him, because heaven knows a teacher wouldn’t ask something completely stupid and obvious to all of us, like what color the sky was…”

    I am an African lady, and I can definitely relate to that line of thought. Back when I was a primary school pupil, I think we (pupils) were very much concerned with impressing the teacher and making high grades than actually learning. Perhaps, learning occurred along the way.

    Being exposed to another educational system in the West was an eye-opener for me, and I still marvel at the difference in systems. I am yet to ascertain which is more effective though, especially at the lower level of education. I see my friend’s younger brothers come home with assignments my little cousins back home could solve at 2 classes below them. But hey, as far as it all goes in and everyone is educated – maybe it is all alright.

    Posted 01 Feb 2011 at 9:23 am

Post a Comment

Your email is never published nor shared. Required fields are marked *