Tuesday, February 26, 2013

Some Things Can't Be Measured


With public education required to “prove” student progress through test scores and number data, it has become more and more common to track the reading progress of MG students through Lexile numbers and other measures of text difficulty, such as Fountas and Pinnell and Accelerated Reader. I have seen these numbers cropping up on Individualized Education Plans (IEPs) for both Gifted and Special Ed students as an official measure of student progress.
The problem is – how do you measure the difficulty of a book? Sometimes, it seems obvious. Take Gary Paulson’s book Masters of Disaster (102 pgs), which is about three boys trying to get into a record book. Compare it to Jane Austen’s Pride and Prejudice. Which one do you think is harder to read?

Pride and Prejudice has a Lexile rating between 900 and 1070, depending on the edition.
Masters of Disaster has a Lexile rating of 1100.

Yes, you read that right.

According to the Lexile website, ratings are measured by a program that looks at word frequency and sentence length.  Hmmm …

First sentence of Pride and Prejudice: It is a truth universally acknowledged that a single man in possession of a good fortune must be in want of a wife.

First sentence of Masters of Disaster: I’ve called you here today, men, because I have an important announcement.

Nope, I’m not seeing it. But there are always flukes. Let’s look at some more.

The Princess Bride – 870. 
Harry Potter and the Deathly Hallows – 980. 
The Hound of the Baskervilles – 1090. 
Vordak the Incomprehensible – 1140.  (Vordak is the “Diary of a Wimpy Kid” of Supervillains.)

I’ve spent a lot of time on the Lexile website, because my daughters are frequently required to select independent reading titles based on a Lexile level assigned to them by a MAP test (which is a whole ‘nother story – don’t get me started!). In sixth grade, one of my daughters was required to select books with a Lexile over 1100 for a reading project every single month. I was ready to pull my hair out, trying to find books she wanted to read – that were actually a challenge for her – and also topped 1100 on the Lexile chart. I didn’t want to have to give her Moby Dick!

The bottom line is: None of these measures – Lexile, F&P, AR – take into account theme, genre, and story complexity. They are all based on formulas, word frequency, and sentence structure (although the results still mystify me). 

Based on my experiences teaching MG readers for a quarter of a century, I can say that – regardless of text difficulty – a book is harder to read if the student is unfamiliar with the setting because it is historical fiction or set in a foreign land. Conversely, fantasy and science fiction books often get a more difficult rating than they deserve because of the made-up names for people, places, and things.  The computer programs don’t know how to handle words like Vordak.

A biography of John F. Kennedy is likely to be harder than a biography of Harry Houdini, even if they are written on the same “level” – because one includes a great deal of politics while the other includes a great deal of escape tricks. Books that include sarcasm and irony harder are harder than books that involve mostly bathroom humor. And finally, a boy who desperately wants to read The Hunger Games because all his friends are reading it will probably have less trouble than a boy handed Little House on the Prairie as a class assignment by his teacher.

Some things ought not be measured by a computer program or mandated in the name of accountability – especially when the measures are superficial and look good on paper while being utterly meaningless. 

The quality and difficulty of a book is one of those things.

20 comments:

  1. Couldn't agree more. We use the AR and STAR test in my district, and it is extremely unreliable. Not one teacher I know finds it accurate. However, since we can't find another program that is reliable, we use AR because the district has vested interest due to the money spent on the program. So I will add that money should not factor in to measuring reading levels either.

    ReplyDelete
    Replies
    1. Mike, you hit the nail on the head there!
      "We bought it. You're using it. End of story."

      Delete
    2. Well, if enough results prove it just doesn't work like they want it to, it won't be the end of the story. Only the footnote to a larger problem.

      That's my HOPE anyway...

      Delete
  2. So agree with you. Judging books so heavily on vague reading skill levels results in weird results sometime. I hope you were able to find books for your daughter that she enjoyed.

    ReplyDelete
    Replies
    1. It was a real challenge at times! We kept having to divert into adult literature to get books with Lexiles high enough, and she was only in 6th grade. It was silly.

      Delete
    2. I can imagine how frustrating that would be. It's hard enough getting kids to read as it is, whether they're reading at grade level or not, this system sounds like yet another way to stack the decks against them.

      This is one area where "logic and math" do more harm than so-called "useless emotion." At least that's how I see it.

      If I have to hear one more story about someone at 13 devouring pre-20th century classics I'm going to lose it!

      I'm 25, and I still find Dickens and Twain beyond me, and I don't consider myself a poor reader, at all.

      Your thoughts, Dianne? Anyone?

      Delete
  3. Right there with you. My impression was that the AR quizzes were intended not to grade students but to check their comprehension to make sure they were reading at the right level. And now they seem to be used in grades.

    I don't like it.

    ReplyDelete
    Replies
    1. The AR quizzes are also very surface-oriented. They just check the child actually read the book, not that she/he understands the themes and character arcs.

      Delete
  4. Don't use these in Max's school thankfully. It's amazing how books can be rated on words without threme, genre, or story complexity. One of the coolest things a teacher told me when Max was in third grade was that fantasy books (because of the complexity of imagery and the need to "see" things outside of the real world) made them a higher reading level. I'd never heard it before though it made sense once I heard it. It also made Max feel like a million bucks because he loved (and still loves) fantasy/sci-fi. But that's the problem with rating scales of all sorts that create a framework but are niether static nor work for every book. Kind of like movie ratings... but don't get me started on G ratings and Disney...

    ReplyDelete
    Replies
    1. It is true that the themes and settings of fantasy and science fiction books often make them harder than their contemporary counterparts. However the use of made-up words in Vordak doesn't make it harder than Pride & Prejudice. It just means computers don't know how to handle Vordak's sinister repertoire of accomplices, nemeses, and weaponry! :D

      Delete
  5. Thankful we don't use AR levels, instead there is a quarterly requirement of 13 books from 9 genre's (poetry, historical fiction, fiction, traditional lit. Etc.,) They also participate in Battle of the Books which includes about 30 books and covers some of the same genre's. it seems to leave more room for choice.

    ReplyDelete
    Replies
    1. I think that's a more reasonable way to assign choice. But the lexiles are used specifically to "chart progress" -- usually to prove significant growth to an outside party, like the state.

      Delete
  6. My daughters both did AR, but they never complained about it, so I never looked into it that much. That said, I completely agree that even the most complicated algorithm would have a hard time "rating" something as subjective as literature.

    ReplyDelete
    Replies
    1. Nope, no mathematical algorithm can address theme, let alone the background knowledge of the reader, etc.

      Delete
  7. How did I not know about this AR garbage until I read this site. Maybe if we payed teachers more, we'd attract better teachers, and we wouldn't need fabricated systems to justify poor educations:)

    ReplyDelete
    Replies
    1. The teachers aren't the ones advocating these book ratings in most cases. It's the government mandated documentation of student progress "by numbers." Lexiles and AR points are just a way to put numbers on paper so that politicians can say, "Okay, that shows learning."

      Delete
  8. I posted about AR here a few year ago, after teaching with the program for a few years. I know plenty of teacher who arrange their library by lexile numbers. I appreciate the sentiment behind both ideas, but would have to agree that the reading experience is more complex than a formula or a set of comprehension questions.

    ReplyDelete
    Replies
    1. Caroline, teachers who buy completely into those programs and blindly follow them make me cringe ...

      And any teacher who has time to look up all the lexiles and file the books in order has too much time on his/her hands!!!

      Delete
  9. I find the lexile levels a useful tools in helping identify books my reluctant reader eight-year-old daughter can actually understand. She was picking books that were too hard for her, and then not really understanding or enjoying them. But I completely agree that it doesn't give the whole picture of a book. There are so many aspects of a story that factor into its accessibility.

    ReplyDelete
    Replies
    1. The lexile levels might be a little more accurate for the early readers, but you still need a human judgment of them. I've seen some books rated lower than they should be, again based on vocabulary and sentence length, when the setting or theme -- or in one case the sarcasm of the main character -- make them harder than the lexile suggests.

      Delete

Thanks for adding to the mayhem!